Throughout 2015 Star Marine, Star Citizen’s first-person shooter module, was always just weeks away. In March, on a stage at PAX East, CEO of Cloud Imperium Games Chris Roberts said it would be released in three weeks. Five months later in August, he told GamersNexus that it was “a matter of three, four, maybe five weeks.” Today, more than a year on, Star Marine still isn’t in players’ hands.
Over the past several months spent researching Star Citizen – you can read the full story of its troubled development so far right here – the story of Star Marine’s development stood out as emblematic of many of the project’s varied problems and challenges over the five-year course of its development so far. The reasons why it’s not playable today come down to the same things that have hit all aspects of Star Citizen’s development: overcoming technical debt, internal management problems, and a reliance on contractors.
Since wrapping up its Kickstarter campaign back in November 2012, Cloud Imperium Games (CIG) has been structuring Star Citizen’s development around the release of modules. The hangar module in 2013, for instance, was intended to show off the fidelity of the game’s ships and environments. 2014’s Arena Commander release got players into ships and dogfighting for the first time. Each module gave players something to play while they waited for the full game, but also pushed CIG’s developers to create the systems needed to support the module.
Star Marine was meant to be the first-person shooter demo. Like Arena Commander, it would take the form of a multiplayer versus game. It would be both an example of what was to come, and a test-bed for developers to get feedback about how well the systems worked and where they could be balanced or fixed.
Work first began on Star Marine back in 2013. CIG had opened its third studio, Foundry 42, based near Manchester, and money from backers was pouring in, with the total pot sitting at around $20 million.
Nonetheless, CIG was stretched. The studio had released the Hangar Module and Arena Commander, both of which it had to support with updates and patches; it was working on what would become the Social Module, an example of an in-game space port for players to explore; and with Foundry 42 up and running, work on Star Citizen’s single-player campaign (Squadron 42) could start in earnest. This was all with less than 50 employees across the company.
One crucial in-game system was taking on growing importance: first-person.
When Star Citizen was originally presented to the public via Kickstarter, Squadron 42 was the core part of the game. It was pitched as a successor to Wing Commander. While players would sometimes venture out of the cockpit, it wasn’t the focus, and so there wasn’t a pressing need to work on first-person systems to cater for it. As Star Citizen’s feature set grew, though, with stretch goals expanding its ambition with each passing month, more and more of the game would rely on first-person systems: boarding ships, going for a space walk, exploring space stations, on-foot shooting.
Roberts didn’t have the staff or the expertise within CIG to handle first-person development internally. This was partly due to the technology underlying Star Citizen, CryEngine, which wasn’t widely used in the industry – meaning there weren’t many developers with experience of the engine. CIG’s Austin studio had also struggled to find qualified staff in an area so saturated with other game developers.
Instead, Roberts hired a contractor to do the lion’s share of the work on Star Citizen’s first-person shooting. From the earliest days of development, CIG had made heavy use of contractors: CIG had used them when building a prototype to show investors, and when CIG was struggling to hire staff at its US studios, the company leaned on contractors to cover the gap. While CIG had only 48 internal staff, in October 2013 it was employing a further 91 contractors.
Roberts sorted a deal with a Illfonic, a studio based in Denver who had made first-person shooters before and had some experience with the CryEngine. (Illfonic chose not to comment officially for this article.)
Work began on the module in the 2013 and the original remit, I’m told by a source who worked on Star Marine, was to produce a vertical slice of Star Citizen’s first-person combat. “It was pretty much what they had with the flight sim, Arena Commander. It was that, but for the FPS: a really small vertical slice of what the bigger picture would have been.”
At the start, plans for the module were ambitious. A team of players would begin as marines in the belly of a ship that would need to fly out to a pirate-owned space station, the Gold Horizon. They’d fight their way to the station, docking at one of its three docking points, and then board the deep space complex. “It was supposed to be this little contained game within in the Star Citizen universe using some of the features of what they wanted to expand on later,” my source explained.
This was later pared down to what was shown 12 months later at PAX in late 2014:
A crew of players would be dropped off in the main (and, now, sole) hangar of the Gold Horizon in a predetermined sequence. It was on them to take the station, and another team of players would try to stop them.
The PAX demo makes it look like there’s a scripted mission, but that was staged to make it more interesting to watch. “If it was a normal versus FPS then no one would think it was cool,” my source explained. “That was just for PAX. They talked about scripted missions but not at the first release. The first release was just 'Here are some levels and here's the FPS gameplay within Star Citizen'.”
Besides that, though, everything that was shown at PAX worked. It was a functional versus FPS mode being played live on stage over a network. The section where the attacking team disables the station’s power core and gravity disappears? That all worked. At that point the plan was to spend the next few months polishing the module and to have it released to players by April 2015.
In February, Roberts was saying in interviews that Star Marine’s release was imminent - but behind the scenes it was a different story. “A lot of things were ready to go and a lot of them were at 85%,” my source says. “CIG didn’t want to pull the trigger.”
What had happened behind the scenes was that as CIG looked into Illfonic’s module, it realised it wouldn’t work with the rest of what had been built for Star Citizen.
“They basically worked separately from the code [of the main game] for about three months and the code paths had diverged so much they didn't merge up well,” Chris Roberts claims. CIG’s lack of internal producers to manage the project contributed to the diversion, he says. “That was our fault for allowing that to happen and not having greater technical oversight.”
However, as Illfonic continued to polish the Star Marine module, CIG started to ask for changes. “Once different people started to see it, they'd have an idea, and then once Chris liked the idea it just had to happen,” my source recalls. “Every couple of weeks they wanted to add something or they wanted to change something and that would erase several months’ work. We tried to hit every delivery and they kept changing it.”
The Gold Horizon map was essentially remade multiple times. Over more than a year of development, new demands and development practices from CIG meant it was constantly being iterated upon and never quite finished. One time it was because when Illfonic first started making Star Marine, CIG's workflow called for extremely detailed environments. “Anyone who has made environments before will know this is a huge resource hog,” a source explained. “You're not going to get a good frame rate. The assets need to be cheap to run to get more effects and lighting in.”
“The whole time the art team was wondering if the map was going to run, if it was going to perform, because CIG wanted incredibly detailed environments early on,” said another person who worked on the project. By the time Illfonic’s work on Star Marine was wrapping up, CIG had adopted more efficient techniques that Illfonic’s leads had suggested in the beginning. This meant all the highly detailed environment assets had to go in favour of ones built with the more efficient method: a necessary change, but a time-consuming one.
The worst example of wasted effort was discovered towards the end of Illfonic’s time on Star Marine: CIG found that the entire map was built to the wrong scale.
CIG builds all its ship and station interiors to an established scale so that each asset can become part of an environment kit. For this to work, a source explained, “you need to have the same slottable pieces for all different types of art styles. All standard doors, for example, whether they be for a moon base or a Mars base, have to share the same dimensions. If you’re building a new environment and new art assets to go with it then you create them as standard, modular pieces so other environments in the same style can be built quickly without needing bespoke assets.”
Unfortunately, the assets that Illfonic had created for the Gold Horizon level did not fit into the levels that CIG had built. CIG asked lllfonic’s artists to remake the lot.
“I'm always very perplexed by this,” Roberts told me when I asked how this happened. “We got everyone together and had a whole art summit in Austin in 2013. I thought we were all on the same page but I guess at some point we weren't.”
It all seems to come down to a lack of producers on an already-stretched team. There was no one person in a position to spot these problems. Months of work had to be redone to fix the scale problem.
It wasn’t just the Gold Horizon map that couldn’t be locked down and completed: the animation team had to retarget all their animations repeatedly throughout the two years Illfonic worked on Star Marine. “The animation team hated it,” a source told me.
Animation rigs were shared across all of the studios working on Star Citizen. These skeletons could be slotted inside a character model and animators could create animations they needed for their specific aspect of the game.
“Once you have a rig with a character you can't really change a lot because you have multiple studios working with the same rig to do animations within their module,” a source elaborated. “If anyone in the core CIG team changed one of those rigs, they needed to retarget every single animation to the new skeleton.” CIG did that “several times”, which was “an absolute pain in the ass for the animation team”.
When I brought this up with Roberts, he explained that internal problems CIG had caused this. When Illfonic started work on Star Marine, they were using a character model that had been created by the Austin character team. However, Roberts tells me, “we weren't particularly happy with the output we were getting in Austin”, so character production was moved to the UK, where CIG “had the opportunity to hire people who had experience with CryEngine”. Roberts explained that this led to the UK team creating a new character model (and the skeleton to go with it) from scanning real-life actors, which was on a slightly different scale to the previous ones. This meant that all the animators had to retarget their animations.
“[There] you have an example of the UK team doing its thing that then has a ripple effect on the Illfonic team, who get upset that they're having to retarget everything, and all that was frustrating,” Roberts says. “That skeleton got redone again because it wasn't right. Now we have this really nice male and female model and that's not changing.” Of course, when the skeleton was redone for the second time, it called for the animators to retarget all their animations yet again.
“That's an illustration of the things that can go wrong with communications,” Roberts continued. “The fact that you can see the animation team bitching about other people, and you've got people at Illfonic bitching about what the character team gave them. So from each side's perspective they probably think the other team are assholes. Me, at the top, and [global production lead] Erin [Roberts] and the other people are trying to stop that from happening and getting people on the same page. That's kind of a challenge and not just for Star Citizen. This happens on all big projects.”
Meanwhile at Illfonic, the work to get the Star Marine it had created to work with the ever-changing Star Citizen was becoming a real burden for the studio. Star Marine was delayed, and delayed again. One source told me that eventually Chris Roberts asked Illfonic if it could reconcile the problems in the FPS module, and the company instead ended its partnership with CIG.
“Illfonic sent the email,” a source tells me. “It was a mutual thing but Illfonic sent the email.”
CIG had already hinted that it wanted to take the first-person module in-house. But “at the same time, [Illfonic] had had enough’,” says a source. “They sent the email, they wanted out.” After two years of working with CIG on Star Marine with no end in sight, and having to repeatedly redo its work, Illfonic’s team morale was shot. “It was numbing,” a source said.
Another source at CIG told me that “Illfonic was producing what they should have been delivering [...] the fault landed on our internal requirements. It's going to be very difficult for any outside vendor to match what we're asking [if] it's a constantly moving target. Couple that with a lot of money being spent on manpower per month and it didn't seem financially feasible to keep them on.”
A press release in August 2015 stated that CIG planned to move development of the module internally.
In the two years since Illfonic began working on Star Marine, CIG had managed to hire significantly more staff. By October 2015, it had more than 250 people working at its studios, compared to just 75 contractors. It had opened a new studio in Frankfurt specifically to hire staff leaving Crytek, the original maker of the CryEngine. CIG’s management felt that it finally had the manpower and expertise to handle the development of Star Marine internally. In 2015 CIG brought everything that Illfonic had made for Star Marine and put it under the direction of the Frankfurt studio.
“Having two thirds of your game out of house to me is a bad idea,” Todd Papy, design director at CIG’s Frankfurt studio, says. “When Chris was building up this company he didn't have somebody who could watch over FPS at that time, which is one of the reasons they reached out to Illfonic and started working on it.”
When Papy first started in 2014 he was having daily Skype meetings with Illfonic, recording himself playing the latest build of the game and providing feedback for the contractor, Papy recalls. “I have videos of myself sitting here playing and saying ‘When you're moving left and right it's not responsive enough, there's lag, you basically feel drunk. We've got to work out gun recoil, and so on and so forth.” One of the issues Papy’s team was having was the time difference – “Denver is eight hours behind” – so there was little crossover when both teams were working.
Meanwhile more and more ex-Crytek staff were joining, many of whom had worked on the Crysis series. Some had been working on Crytek shooters as far back as Far Cry. “We started to hire more people in-house to build up the team,” Papy says. “Once it was ready to pull in house, we pulled it in house.”
When that happened, CIG discovered that most of the code base Illfonic had created for Star Marine would have to be rewritten to make it work with the rest of Star Citizen.
“We basically had to refactor everything to work how the big world works,” Roberts said. “If we were just doing a standalone FPS shooter then, yeah, we could probably have just packaged up the Illfonic Star Marine stuff they worked on and released that, but that wasn't the point of it. It has to be seamless. It has to be everything – getting in and out of ships, flying around, going to different locations. They [Illfonic] were doing some things that would be fine for a running around, Counter-Strike style shooter but that doesn't work with what we were doing.
“As I said, at the time we didn't have the resources so we kind of said 'Well, they can do this.' But we would do it differently now. [...] If more time was around then probably we wouldn't have brought [a contractor] on, but at the time we were like 'We've got to get the FPS working, we've got to get space flight working. We have the money so let's get the resources working.'”
“Initially, because Chris wanted to get things delivered so quickly, he went to a lot of different outsourcers,” Erin Roberts, global head of production, told me. “It did become quite apparent early on that we weren’t getting the kind of results [we wanted], either technically or visually, and it would need so much more extra bandwidth.”
“If you outsource something, it has to be really controlled and direct,” Nick Elms, creative director at Foundry 42, added. “It doesn’t matter how good your outsourcers are, the stuff will come back and you’ll be spending a lot of time yourself fixing it so it works the way you need it. At that point you’re better off paying the money to just have the resources internally working within the team and getting those skills trained up internally.”
Since moving development of Star Citizen’s first-person systems internally work on it has progressed swiftly, CIG’s staff told me.
“The way that the animations are broken down and the way we're doing the camera a lot of this required extremely low level knowledge of the CryEngine. Illfonic didn't have that information, unfortunately,” Todd Papy told me. Now that the module has moved internally it’s much easier. “In the UK – that's where our programmers and animators are based – they're only an hour behind us. So it's like a sister studio: we talk on Skype, download the build, play it, call each other up. The iteration time has sped up a hell of a lot.”
You can hear about the experimentation that has gone into getting Star Marine’s first-person camera working in the video below:
There are still some contractors working with CIG today. “Behavior still does a bit of work for us, [and] there's a UK company too called Airship that's doing heads for our characters, that's kind of a specialist thing,” says Roberts. “But I'd say the majority of our stuff is created by the staff, which is one of the reasons we shifted headcount. We've also had some not great experiences of trying to outsource to China, with language barriers and turnaround issues... We just ran the numbers and said 'It's more effective and more efficient [to do it ourselves].'
Something else that came into effect after the formation of CIG is the UK’s game tax credit. “That instantly made everything in the UK, like 25% cheaper,” Roberts said “So, once you add in the markup, the overhead, and the management overhead, it's actually cheaper for us to do the work internally than it is to outsource it to China. Basically, It's one of the reasons why the studio [in England] has grown as much as it has.”
Tony Zurovec, CIG’s director of the persistent universe in Austin, does defend the earlier use of third-parties. “It sounds a bit crazy, but it’s true: even when you've four different studios in cities around the world, you always can't find people willing to go to those places,” he said. “There's a large number of people that you can't get. If you look in the early stages of CIG's existence, the large majority of the environments were being developed at Behavior at that time. Behavior got up to almost 50 contractors there for a while, and this was when CIG was 220, 240 people. That one contractor was a quarter of our entire manpower.”
“As we had the capability to bring all that stuff in-house so we could control it more precisely, as we got the leadership in the position to where we could have somebody whose full-time job is to worry about that one single element [of the game], that's a dramatic change from where the company was two years ago.”
Throughout much of 2016 Roberts and CIG staff avoided talking about Star Marine, prompting speculation that it might have been canned. For backers this was a shock, especially as throughout 2015 it was talked about as if it were mere weeks away. Then, during a 50-minute presentation at Gamescom this past August, Roberts announced that not only was Star Marine coming, it would be released in the next major update: Alpha 2.6.
CIG also released another video of Star Marine in action, showing off all the improvements made by its in-house developers:
Star Marine is emblematic of the challenges CIG has faced over the past five years. Its development was stymied by staff shortages, production problems and significant engineering work. CIG believes it has overcome these problems and that it is finally making rapid meaningful progress.
However, we’ve been here before, and there still isn’t a release date for Alpha 2.6. It may be shortly after CitizenCon, the annual Star Citizen convention that is taking place on October 9th. But that’s speculation for now.
This is part of an ongoing series about Star Citizen on Kotaku UK. Read our investigation into the game's development so far, Inside The Troubled Development of Star Citizen, right here, and see the rest of the series here.