Ok so this is to be a semi brief overview of the networking protocol and related systems as we currently under stand them.
This is mirrored on the wiki at: https://github.com/themeldingwars/Documentation/wiki and is here for easier discussion and feedback.

Login and Zone Selection

This will just be a brief overview of this part, there is more to this that can be covered later.

The initial steps are done over HTTPS to a web server.

  • User sends a log in request with user name / password
  • If authenticated the web server sends back a character list, this gets you to the choose a character screen.
  • When the user clicks the "Enter World" button another HTTPS request is made to get an "Oracle Ticket"
  • The server sends back an oracle ticket like below.
"matrix_url": 127.0.0.1,
"ticket": "A base64 encoded data string, contents not known as of yet",
"datacenter": "localhost",
    "operator_override": {
        "ingame_host": "https://indev.themeldingwars.com/ingame_host",
        "clientapi_host": "https://indev.themeldingwars.com/clientapi"
    },
    "session_id": "8360c86c-a3d0-11e4-9e16-c074c1266b6d",
    "hostname": "localhost",
"country": "GB"

The key bit here is the matrix_url this is the ip that the game client will now start sending packets to over UDP on port 25000

Note: We have a simple test web API up for local server testing at https://indev.themeldingwars.com this server just returns an Oracle ticket pointed to 127.0.0.1:25000

To use open your Firefall.ini in the games install and add

[Config]
OperatorHost= "indev.themeldingwars.com"

Game Server Connection

The data is in Big Endian

Handshake

Now that the client knows where to connect it has to establish a connection with that game server.
This is all done over UDP and the game uses a custom networking protocol over UDP for reliability when it is deemed necessary.
The command format for these handshake messages is:

[uint]         [char[4]] 
----------     -----------
SocketID        Id for msg    
00 00 00 00     50 4F 4B 45

For the matrix handshake the socket id is always 0
The first packet that the client sends a POKE message to the server to start off the connection.
(You can use matrix_connectiontest ip in the console to test the handshaking and a simple connection)

The flow is

[Client]          [Server]
POKE        ->
            <-     HEHE
KISS        ->
            <-     HUGG

(Yes they really are called HEHE, KISS, POKE and I love it :D) (and aperantly a hug comes after the kiss)

Below are the packets send and received as part of the handshake.

POKE (Client)

[uint]         [char[4]]      [uint]
----------     -----------    ----------------
SocketID        Id for msg    Protocol Version
00 00 00 00     50 4F 4B 45   00 04 B9 68

HEHE (Server)

[uint]         [char[4]]      [uint]
----------     -----------    ----------------
SocketID        Id for msg    SocketID
00 00 00 00     50 4F 4B 45   F1 07 12 8E

KISS (Client)

[uint]         [char[4]]      [uint]              [ushort]
----------     -----------    ----------------    ----------------
SocketID        Id for msg    SocketID            Streaming Protcol
00 00 00 00     50 4F 4B 45   F1 07 12 8E         4C 5F

HUGG (Server)

[uint]         [char[4]]      [ushort]            [ushort]
----------     -----------    ----------------    ----------------
SocketID        Id for msg    SequenceStart       GameServerPort
00 00 00 00     50 4F 4B 45   F7 A1               D2 65

ABRT

[uint]         [char[4]] 
----------     ----------- 
SocketID        Id for msg  
00 00 00 00     41 42 52 54

Used to end a connection

Main Connection

After a successful handshake the client will then start to communicate to the game server on the port that was assigned in the above handshake. eg the game will start sending packets to 127.0.0.1:53861 and the client will receive packet on port that the first matrix handshake message was received from.

All packets to and from the server start with a uint for the socket id that was assigned from the handshake. (Again if the id is 0 is a matrix handshake message).

Below is the packet header:

[uint]         [2bits]      [2bits]         [1bit]         [11bits] 
----------     ---------    ------------    ------------    ------------ 
SocketID       Channel      Resend Count    Is Splitted     Length

TODO: Check

UDP packets have a max size and so if a message being sent is bigger than this size it will have to be split across multiple packets.
If a packet is splitted then the Is Splited will be 1, split packets are per channel and not global across them.
If a channel comes across a split packet then that stream is blocked until all the all the fragments for it are received and can be assembled back into the full packet.

If a packet is marked as splited the data for that should be held in a buffer and the next message with the sequence number directly after should be appended. This process should be repeated until the next packet in sequence that doesn't have the Is Splitted set is encountered, this marks the end of that split packet.
All the buffered packet data up untill but not including this should be reassembled to form the full packet data.

There are 4 different channels that a message can get sent over Channel indicates what one the message is using.
Channel 1 and 2 are reliable while channel 0 and 3 are unreliable.

We need to branch the logic for parsing a packet here to handle the channel types.

Channel 1, 2 and 3

All of these channels have another ushort for the sequence number for that packet.
The sequence number is per channel and per direction. eg sending on channel 1 will have a different sequence number than sending on channel 2.

Channel 1 and 2

Since these are both reliable channels we need to check the Resend Count, if this is greater than 0 than we need to xor the data to get the correct values. and ofc its not a static value to xor against, we use the Resend Count to index a preset list of xor values.
Below is the list of xor values to use.

byte[] xors = new byte[] { 0x00, 0xFF, 0xCC, 0xAA };

eg if the Resend Count is 1 then we would XOR all the data after the Sequence Number with 0xFF (xors[Resend Count])

Channel 2 and 3

The next part of the header is common for both of these channels.

[byte]           [byte[6]]    [byte] 
-------------    ---------    ------------
Controller ID    Entiy ID     Msg ID

Note: This is the same as the player CID that can be gotten from the LUA API only with the first and last byte overwritten.

The data after this header is packet specific.
We can route to the correct handler with the controller ID and the Msg ID
Note: At this point the data here is very similar to that contained in the replay files

Below is the way to route the packet handling:

click to show

Channel 0

Channel 0 seems to be for protocol level control commands such as ACK's, NACK's and time updates.

After the previous packet header channel 0 only adds a byte as the message id, after this is per packet data.
I won't list some of the channel 0 packet formats here to save some space.

Channel 1

Channel 1 just adds a byte for the message id after the previous header and everything else after that is per packet data.
These packets seem to be for the more abstarct game functions that don't directly relate to in world entitys.

Below is the packet names for each known packet:

click to show

Channel 3

If a message sent in this channel get too long and needs to be split it will instead be split up and sent in channel 2 as a reliable packet. (The client emits warning in cases like this)

Channel 3 is much like channel 2 but unreliable.

Packet parsing overview diagram

0_1560222154078_Firefall Network Proto.png

Todos and whats next:

  • Verify some of the assumptions that are made above.
  • Iron out some of the smaller details, eg handling droped packets.
  • Finish libary for handling the low level protocol stuff so we can focus on the packet contents and have a good framework for writing parsers for them.
  • Documentation for for those packet formats.
  • More tooling for exploring packet captures nicely (PacketPeep and 010 Buddy).
  • Formating and feed back on restructing this so that it is easyier to read and understand.