At some point your piping becomes too small and starts to be a restriction. How much of a restriction? Who really knows(math). Personally I say match it to your TB size for as much of your charge pipe intact tract as you can. This is a general 'rule of thumb'. In our case with the n54 the TB is 76MM or 2.99''. I would go with an IC with 3'' in and out and 3'' CP from IC to TB. I see no reason to have a restriction at the IC inlet albeit it may be small, but the faster you get to 3'' the better imo. That also happens to be exactly what I run

I step up from the Precision 6466 outlet to 3'' over 1 foot of pipe that then connects to my 3'' inlet on my IC(VRSF Race 7.5'') inlet.
According to Corky Bell, Maximum Boost page 61: 304 MPH or 0.4 mach is the point at which airflow meets increased resistance (drag) and flow losses are experienced as well as increased turbulence, if I recall correctly(might be off by 1 page)
0.4 mach = 304 MPH
2" piping
1.57 x 2 = 3.14 sq in
300 cfm = 156 mph = 0.20 mach
400 cfm = 208 mph = 0.27 mach
500 cfm = 261 mph = 0.34 mach
585 cfm max = 304 mph = 0.40 mach
2.25" piping
3.9740625 sq in = 1.98703125 x 2
300 cfm = 123 mph = 0.16 mach
400 cfm = 164 mph = 0.21 mach
500 cfm = 205 mph = 0.26 mach
600 cfm = 247 mph = 0.32 mach
700 cfm = 288 mph = 0.37 mach
740 cfm max = 304 mph = 0.40 mach
2.5" piping
4.90625 sq in = 2.453125 x 2
300 cfm = 100 mph = 0.13 mach
400 cfm = 133 mph = 0.17 mach
500 cfm = 166 mph = 0.21 mach
600 cfm = 200 mph = 0.26 mach
700 cfm = 233 mph = 0.30 mach
800 cfm = 266 mph = 0.34 mach
900 cfm = 300 mph = 0.39 mach
913 cfm max = 304 mph = 0.40 mach
2.75" piping
5.9365625 sq in = 2.96828125 x 2
300 cfm = 82 mph = 0.10 mach
400 cfm = 110 mph = 0.14 mach
500 cfm = 137 mph = 0.17 mach
600 cfm = 165 mph = 0.21 mach
700 cfm = 192 mph = 0.25 mach
800 cfm = 220 mph = 0.28 mach
900 cfm = 248 mph = 0.32 mach
1000 cfm = 275 mph = 0.36 mach
1100 cfm max = 303 mph = 0.40 mach
3.0" piping
7.065 sq in = 3.5325 x 2
300 cfm = 69 mph = 0.09 mach
400 cfm = 92 mph = 0.12 mach
500 cfm = 115 mph = 0.15 mach
600 cfm = 138 mph = 0.18 mach
700 cfm = 162 mph = 0.21 mach
800 cfm = 185 mph = 0.24 mach
900 cfm = 208 mph = 0.27 mach
1000 cfm = 231 mph = 0.30 mach
1100 cfm = 254 cfm = 0.33 mach
1200 cfm = 277 mph = 0.36 mach
1300 cfm max= 301 mph = 0.39 mach
Then you go here as an example to do some math:
Promotional website for the Water-Cooled VW Performance Handbook, Third edition, by Chad Erickson and Greg Raven.
www.gregraven.org
Read more about it here:
https://www.dsmtuners.com/threads/how-much-will-different-sized-intercooler-piping-flow.356270/
This explains it ok, but rather simplified:
You can do all the math and calcs etc. to find out how much you need at a given HP. Or just go 3'' and be done. 3.5 or 4'' is overkill for the N54 and would be a PITA to fit. There is also the limitation of the stock Tb at 3''. 3'' is as large as you can go without cutting things up and getting creative. People often use too small piping and get away with it mind you, but it is not 'ideal'.
In theory larger piping takes longer to pressurize. In real life you are not going to notice it. Unless you went from something WAY too small to something too large. like 2'' to 4'' over the entire intake tract after the turbo(s).
In other words: Give her the ol 3 inches!
Edit: All this said you can get by running a larger turbo, having more AKI, etc. Plenty of people make good HP on other platforms, and ours, with 2.5'' and even smaller. Plenty of platforms make monster power on a 3 or 3.5'' LS for instance. But when it is easy to run 3'' for most of the charge piping on an n54, why not? The cost difference is minimal.
Edit #2: Yeah, get outlets.