To use custom payloads using PS3JB, you need to convert a binary file of the payload data to a TI-84 Plus/Silver Edition AppVar (*.8xv file), transfer it to your calculator, and specify in the application's options that you want to use this payload.
This is broken up into four steps: obtain the payload data, convert it to an AppVar, transfer it to your calculator, and select it in PS3JB's options.
The PS3JB exploit works by transmitting a "stage 1" payload to the PS3 via the configuration descriptor of the first device attached to the exploit's virtual hub.
The payload AppVar used as "stage 1" of the exploit is essentially this configuration descriptor. If you've never seen a configuration descriptor before, the first few bytes usually resemble the below's:
Some PSGroove/PSFreedom ports may specify this data in a header (*.h) file, or it might specify them in a binary (*.bin) file, which is preferred.
It's also possible you'll find payloads that don't start with the configuration descriptor (09 02 12 00 01 ...). If this is the case, you'll need to tack on the descriptor at the beginning before moving on.
An example of a binary file of payload data is the official 1.1 PSGroove payload here. Notice it has been padded to 3840 bytes, as the exploit requires.
Once you've found/created a binary file of the configuration descriptor/payload, you'll need to convert it to an AppVar that the calculator can receive and store. You can do this with the To8XV.exe command-line application here. Download the application and unzip it to a folder, say on your desktop.
Then open a DOS command prompt (Start | Run..., type "cmd.exe", and hit [ENTER]) and navigate to the folder containing To8XV.exe using the "cd" (change directory) command.
Now enter the following command:
To8XV.exe [payload file name] [max 8-character filename on calculator]
where [payload file name] is the name of the binary file containing the descriptor and payload data and [max 8-character filename on calculator] is the name you want to appear on the calculator.
This will produce a .8xv file for transferring to your calculator.
Transfer this newly-created AppVar to your calculator the same way you transferred the PS3JB application itself.
Start the PS3JB application on your calculator and select 5) Options.
Press the [LEFT] and [RIGHT] keys while "Stage 1 Payload" is highlighted until you see the AppVar you selected and then press [Y=] to save your settings. The application will now use this as the stage 1 payload.