Many guests

From PrgmrWiki

so, on our smaller-spec DomUs, we need to cram a whole lot on to each 32GiB ram/8 core box. here is how to get xen to run with more than around 100 domains:


note, uh, all this is stated in this thread: http://lists.xensource.com/archives/html/xen-devel/2006-12/msg00322.html


Please note, we did this a long time ago, and lost the documentation, and the xenolinux source has changed considerably since 2006, so this might be completely broken. I'll test and report back.


watch yr progress with

[root@localhost ~]# grep Dynamic-irq /proc/interrupts | wc -l
46

edit:

./linux-2.6.18-xen.hg/include/asm-x86_64/mach-xen/irq_vectors.h:#define NR_DYNIRQS		1024

and it works great. this is what we do for the prod. xen 3.4 boxes.

and doing the same on xen4/linux-2.6-pvops

That's all well and good, but we've gotta keep up with the times. If we want to continue receiving free help, we've gotta move to xen4.

So, first thing's first... I tried resetting both NR_IRQS and arch_probe_nr_irqs() to 4096, and testing as per http://lists.xensource.com/archives/html/xen-devel/2010-04/msg01538.html

so, at first, no effect. then I add nr_irqs=2048 to the kernel command line, and now, instead of refusing to start new domains, the thing panics.

[  676.588197] Kernel panic - not syncing: No available IRQ to bind to: increase nr_irqs!

Clearly, I am doing something wrong. Next, I spot this post: http://lists.xensource.com/archives/html/xen-devel/2010-04/msg01537.html

so now, I'm rebuilding the kernel again, increasing NR_DYNAMIC_IRQS in arch/x86/include/asm/irq_vectors.h.


--- ./linux-2.6-pvops.git/arch/x86/include/asm/irq_vectors.h.old        2010-08-11 15:40:53.000000000 +0000
+++ ./linux-2.6-pvops.git/arch/x86/include/asm/irq_vectors.h    2010-08-11 15:41:39.000000000 +0000
@@ -167,7 +167,7 @@
 # if defined(CONFIG_X86_IO_APIC) && defined(CONFIG_SPARSE_IRQ)
 extern int nr_dynamic_irqs;
 # else
-#  define NR_DYNAMIC_IRQS                       256
+#  define NR_DYNAMIC_IRQS                       2048
 # endif
 #endif

also the .config needs

CONFIG_SPARSE_IRQ=n 

(this is being tested now)

This

for ((vpsnum=50; $vpsnum<60; vpsnum++)); do sudo sh domucreate.bash asdf${vpsnum}jklm 128 3072 172.17.0.$vpsnum; done

is what we used to create many domUs on the test dom0.