How big could a fabber get? Could a fabber build a boat or a house? Here are two big CNC machines, one of which is claimed to do work on boat hulls.
A fabber placing individual drops of building material would be awfully slow for a very large project. One work-around would be to trade away spatial resolution, and let the fabber lay down big handfuls of wet concrete.
Maybe you'd want many fabbers feeding small pieces to an assembler that assembles them into bigger pieces. The assembler must be able to make the small pieces stick together, by gluing them or melting the sides or by using mechanical fasteners such as screws or nuts and bolts. It's possible that the big pieces might then be assembled into very big pieces, and again an assembling machine must be fed from many sources. The assembler would need to be very smart to recognize and correct assembly errors, and would probably need machine vision. This would work well for products from a factory, but might be unsuitable for a house.
A google search for "robot bricklayer" turns up a few modest research efforts. I would have imagined something like the big XYZ stage above with a brick-lifting robot arm, wheeled into position over the site of the future house, but the most advanced effort visible on the web is a standard industrial robot arm picking up bricks instead of doing whatever else robot arms normally do. The arm can't move around the entire volume of the future house, it's not on any sort of XYZ stage, it's just bolted to the floor like any other industrial robot arm. So robotic house construction is still quite a ways off.