I have built similar networks, but it's been years since the last one. So, I'm not really a network guy, but I did once have practical experience
If you use more than one wired port on any access point or modem, expect poor performance.
Most wireless access points and modems that have multiple wired ports have truly horrible switching capabilities, even if the ports are claimed to be gigabit. Packet rates are low, and maximum bandwidth over all ports is rarely more than what a single port can sustain.
El-cheapo unmanaged switch use off-the-shelf chips that can switch 5 to 8 gigabit ports at full speed and maximum packet rates. Of course, you'd still need a router/firewall (between the internet and the LAN), and if you need to segregate internal traffic you'll need a more complex topology.
Here's my suggestion using consumer-grade stuff:
If your internet connection relies on a modem, you can use that as the router, if it has the necessary firewall and routing capabilities.
│ Router │
║ └───┤ Access point(s) │
║ ╔═══╡ Server(s) │
│ Switch │
│ │ │
│ │ │
│ │ │
Local traffic that crosses only the dumb switch is very fast, but is unfiltered. (This means, local server access and access between workstations is very fast, but unfiltered and unmonitored.)
Only traffic to/from internet or wireless access points cross the router. In both cases the router is needed to filter the traffic anyway. Since major traffic (like file server access) never reaches the router, the router is not overburdened, and does not limit traffic speeds.
Access points should be connected to the router, so that the router can limit and filter the wireless access to the internal network.
The double lines show where the most traffic flows. You might wish to monitor the traffic rates there, and trunk multiple ports together to get faster speeds if these become the bottleneck.
If you wanted to segregate traffic -- say, bean counters demand that their traffic is not visible at all to other machines (a common requirement in government offices here), you simply add another dumb switch to the router. Just remember, traffic over a dumb switch is not filtered (and machines connected to the same dumb switch and spoof each other without the router knowing about it).
If you don't need traffic segregation, but have something like two dozen wired machines, just make sure your dumb switch has the necessary switching capability.
While a suitable dumb 8-port gigabit switch only costs about 30€ here, 16 to 24-port ones cost about 150€ to 200€. Larger ones tend to be managed (have routing capabilities), and I'd expect to have to pay 500-600€ for a good 48-port gigabit switch.