Programming an XCORE tile with C and lib_xcore#
The XCORE compiler (xcc
) supports targeting the XCORE using GNU C or C++. A
C platform library lib_xcore provides access to XCORE-specific hardware
features. Lib_xcore is available as a collection of system headers when using
the XCC C compiler; the names of the headers all begin xcore/
.
Parallel execution#
The following code listing shows a multicore “hello world” application using
lib_xcore. Note that, as usual, the program has a single main
function as
its entrypoint - main
starts on a single core and lib_xcore must be used to
start tasks on other cores.
1#include <stdio.h>
2#include <xcore/parallel.h>
3
4DECLARE_JOB(say_hello, (int));
5void say_hello(int my_id)
6{
7 printf("Hello world from %d!\n", my_id);
8}
9
10int main(void)
11{
12 PAR_JOBS(
13 PJOB(say_hello, (0)),
14 PJOB(say_hello, (1)));
15}
PAR_JOBS
#
1#include <stdio.h>
2#include <xcore/parallel.h>
3
4DECLARE_JOB(say_hello, (int));
5void say_hello(int my_id)
The PAR_JOBS
macro, from xcore/parallel.h
, is a lib_xcore construct
which makes two or more function calls, each on its own core. Each invocation of
the PJOB
macro represents a task - the first argument is the function to
call, the second the argument pack to call it with. Note that a function called
by PAR_JOBS
must be declared using DECLARE_JOB
in the same translation unit
as the PAR_JOBS
- the parameters are the name of the function and a pack of
typenames which represent the argument signature. Tasks always have a void
return type.
1 PAR_JOBS(
2 PJOB(say_hello, (0)),
3 PJOB(say_hello, (1)));
When PAR_JOBS
is executed, each of the ‘PJobs’ is executed in parallel, with
each one executing on a different core of the current tile. There is an implicit
‘join’ at the end of this, meaning that execution does not continue beyond
PAR_JOBS
until all the launched tasks have returned. Due to each PJob
running on its own core, at the point the PAR_JOBS
executes there must be
enough free cores available to run all its jobs - this means that there is a
hard limit on the number of jobs which may run, which is the number of cores on
the tile. In the case that some cores are already busy running tasks, the number
of jobs which can run will be fewer. In the case that there are not enough cores
available, the PAR_JOBS
will cause a trap at runtime.
Stack size calculation#
In the previous examples, multiple tasks were launched by a single thread, and
yet it was not necessary to specify where the stacks for those threads should be
located. This is because the PAR_JOBS
macro allocates a stack for each
thread launched, as a ‘sub stack’ of the launching thread’s stack. The size of
the stack allocated is calculated by the XMOS tools, which also ensure that the
stack allocated to the calling thread is sufficient for all its launched threads
and callees.
It is possible to view the amount of RAM which has been allocated to stacks
using the -report
flag to the compiler:
$ xcc -target=XCORE-AI-EXPLORER hello_world.c -report
Constraint check for tile[0]:
Memory available: 524288, used: 22408 . OKAY
(Stack: 1620, Code: 19528, Data: 1260)
Constraints checks PASSED.
To enable this calculation, the compiler adds special symbols for each function which describe their stack requirements. These symbols are detailed in the XCore ABI document. The tools are not always able to determine the stack requirement of a function - such as when it calls a function through a pointer, or when it is recursive. In the case that the compiler cannot deduce the stack size requirement of a function, it is necessary to provide a worst-case requirement manually. This is possible either by defining the symbols manually, or by annotating the code (this is discussed later in this document).
Hardware timers#
Timers are a simple resource which can be read to retrieve a timestamp. Timers
can be configured with a trigger time, which causes the read operation to block
until that timestamp has been reached. This makes timers suitable for measuring
time as well as delaying by a known period. Timers can be accessed using
lib_xccore’s xcore/hwtimer.h
- this provides a number of functions for
interacting with timer resources. Like all resource-related functions,
lib_xcore’s timers work on a resource handle. As timers are a pooled resource,
the XCore keeps track of available timers and allocates handles as required.
hwtimer_alloc
allocates a timer from the pool and returns its handle. As
there is a limited number of timers available, the allocation can fail - in
which case it will return 0.
1#include <stdio.h>
2#include <xcore/hwtimer.h>
3
4int main(void)
5{
6 hwtimer_t timer = hwtimer_alloc();
7
8 printf("Timer value is: %ld\n", hwtimer_get_time(timer));
9 hwtimer_delay(timer, 100000000);
10 printf("Timer value is now: %ld\n", hwtimer_get_time(timer));
11
12 hwtimer_free(timer);
13}
14
Channel communications#
The standard way of allowing tasks to communicate is to use the XCORE’s ‘chanend’ resources to send data through the communication fabric. This has the advantage of allowing synchronised transactions even with tasks running on a different tile or package.
Channels#
Lib_xcore provides a channel type in xcore/channel.h
which is used to
establish a channel between two tasks on the current tile. The following code
listing is a program where the sends_first
task sends one word of data to
receives_first
, which responds with a single byte. In this program,
channel_alloc
is called to establish a channel which can be used by two
tasks to communicate. Notice that the returned channel_t
is simply a
structure containing the two ends of the channel. The channel is created before
the tasks are started, and each task is passed a different end of the channel.
1#include <stdio.h>
2#include <xcore/channel.h>
3#include <xcore/parallel.h>
4
5DECLARE_JOB(sends_first, (chanend_t));
6void sends_first(chanend_t c)
7{
8 chan_out_word(c, 0x12345678);
9 printf("Received byte: '%c'\n", chan_in_byte(c));
10}
11
12DECLARE_JOB(receives_first, (chanend_t));
13void receives_first(chanend_t c)
14{
15 printf("Received word: 0x%lx\n", chan_in_word(c));
16 chan_out_byte(c, 'b');
17}
18
19int main(void)
20{
21 channel_t chan = chan_alloc();
22
23 PAR_JOBS(
24 PJOB(sends_first, (chan.end_a)),
25 PJOB(receives_first, (chan.end_b)));
26
27 chan_free(chan);
28}
This approach has the advantage that the two tasks are decoupled - as long as they implement the same application-level protocol, neither one needs knowledge of what it is communicating with or where the other end of the channel is executing within the network.
Within the tasks, chan_
functions are used to send and receive data. These
functions synchronise the tasks since chan_out_word
will block until
chan_in_word
is called in the other task, and vice versa. It is imperative
that the correct corresponding ‘receive’ function is called for each ‘send’
function invocation, otherwise the tasks will usually deadlock or raise a
hardware exception.
Streaming channels#
Regular channels implement handshaking for each send/receive pair. This has the advantage that it synchronises the participating threads, and additionally prevents tasks progressing if more or less data is sent than was expected by the receiving end. Whilst handshaking is desirable in most cases, it does incur a runtime penalty - in the time taken to perform the actual handshake, and often by the sending task being held up by synchronisation.
Streaming channels are provided to address this. For each chan_
function in
xcore/channel.h
, there is a corresponding s_chan_
function in
xcore/channel_streaming.h
. These have the same functionality as their
non-streaming counterparts, except that handshaking is not performed. Each
chanend has a buffer which can hold at least one word of data. Sending data over
a steaming channel only blocks when there is insufficient space in the buffer
for the outgoing data.
Routing capacity#
The communication fabric within and between XCore packages has a limited capacity meaning that there is a limit to the number of routes between channel ends which can be in use at any given time. If no capacity is available in the network when a task attempts to send data, that task is queued until capacity becomes available. Regular channels establish and ‘tear down’ a route through the network on each transaction, as a side-effect of the handshake. This means that routes remain open for very limited times, and all tasks have an opportunity to utilise the network. However, as streaming channels do not perform handshaking, their routes remain open for the entire duration between their allocation and deallocation. If too many streaming channels are left open, this can starve other tasks of access to the network (this includes tasks using non-streaming channels). For this reason, it is advisable to leave streaming channels allocated for as little time as possible.
Basic port I/O#
Ports are a resource which allow interaction with the external pins attached to an XCore package. Each tile has a variety of ports of different widths; these will often have pins in common, and the actual mapping of ports to pins varies by package. Ports are extremely flexible and can be used to implement I/O peripherals as software tasks.
Unlike pooled resources, ports are not allocated by the XCore (as a port mapped
to the desired pins will always be required). Instead, port handles are
available from platform.h
and have the form XS1_PORT_<W><I>
where
<W> is the width of the port in bits, and <I> is a single letter to
differentiate the port from others of the same width. E.g. XS1_PORT_16A
is
the first 16 bit port. As ports are not managed by a pool, they must be enabled
explicitly before use, and disabled when no longer required.
Like timers, Ports have a configurable trigger which, when set, causes reading
the port to become blocking until the trigger condition is met. For example, it
is possible to configure a port so that input is blocking until its pins read a
particular value. By default, this trigger persists so input will be
non-blocking whilst the condition is met but will become blocking again once the
trigger condition becomes false. Functions for interacting with ports -
including input, output and configuring triggers - are available from
xcore/port.h
.
The following program lights an LED while a button is held. Each time the port is read, its trigger condition is updated to occur when its value is not equal to its current one - this means the port is only read once each time the value on its pins changes.
1#include <platform.h>
2#include <xcore/port.h>
3
4int main(void)
5{
6 port_t button_port = XS1_PORT_4D,
7 led_port = XS1_PORT_4C;
8
9 port_enable(button_port);
10 port_enable(led_port);
11
12 while (1)
13 {
14 unsigned long button_state = port_in(button_port);
15 port_set_trigger_in_not_equal(button_port, button_state);
16 port_out(led_port, ~button_state & 0x1);
17 }
18
19 port_disable(led_port);
20 port_disable(button_port);
21}
Event handling#
Events are an important concept in the XCore architecture as they allow resources to indicate that they are ready for an input operation. Lib_xore provides a select construct for waiting for an event and running handling code when it occurs.
1#include <platform.h>
2#include <xcore/port.h>
3#include <xcore/select.h>
4
5int main(void)
6{
7 port_t button_port = XS1_PORT_4D,
8 led_port = XS1_PORT_4C;
9
10 port_enable(button_port);
11 port_enable(led_port);
12
13 SELECT_RES(
14 CASE_THEN(button_port, on_button_change))
15 {
16 on_button_change: {
17 unsigned long button_state = port_in(button_port);
18 port_set_trigger_in_not_equal(button_port, button_state);
19 port_out(led_port, ~button_state & 0x1);
20 continue;
21 }
22 }
23
24 port_disable(led_port);
25 port_disable(button_port);
26}
The above application shows a lib_xcore SELECT_RES
, from xcore/select.h
,
which handles a single event. This is equivalent the example in
Basic port I/O, except that the port_in
will not
block since it is not executed until the port’s trigger condition is met
(instead, the thread will block at the start of the SELECT_RES
).
In this example, and previous examples in this document, the task blocks waiting for a single resource. Conceptually this is often a good design, and is the model generally encourage in the programming model. However, this is generally not an effective use of the available cores - most tasks will be paused in a wait state most of the time. It is also possible that a task needs to be able to accept input from multiple resources (e.g. a ‘collector’ task which listens to multiple channels).
For these reasons, the XCore allows a task to configure multiple resources to generate events, and then wait for any one of those events to occur. The lib_xcore select construct supports this - making it analogous to a Unix select. Conceptually, the XCore multiplexes events which are of interest to a task, and the select demultiplexes them.
![../../../_images/multiplexing.png](../../../_images/multiplexing.png)
The following listing introduces a timer which allows the LED to flash whilst the button is held. If an event occurs whilst another is being handled, the later event remains available on its respective resource and will be handled once the running handler continues.
1#include <platform.h>
2#include <xcore/hwtimer.h>
3#include <xcore/port.h>
4#include <xcore/select.h>
5
6int main(void)
7{
8 port_t button_port = XS1_PORT_4D,
9 led_port = XS1_PORT_4C;
10
11 hwtimer_t timer = hwtimer_alloc();
12 port_enable(button_port);
13 port_enable(led_port);
14
15 int led_state = 0;
16 unsigned long button_state = port_in(button_port);
17 port_set_trigger_in_not_equal(button_port, button_state);
18
19
20 SELECT_RES(
21 CASE_THEN(button_port, on_button_change),
22 CASE_THEN(timer, on_timer))
23 {
24 on_button_change:
25 button_state = port_in(button_port);
26 port_set_trigger_value(button_port, button_state);
27 continue;
28
29 on_timer:
30 hwtimer_set_trigger_time(timer, hwtimer_get_time(timer) + 20000000);
31 if (~button_state & 0x1) {
32 led_state = !led_state;
33 port_out(led_port, led_state);
34 }
35 continue;
36 }
37
38 port_disable(led_port);
39 port_disable(button_port);
40 hwtimer_free(timer);
41}
The SELECT_RES
macro takes one or more arguments, which must be expansions
of ‘case specifiers’ from the same header. The most common case specifier is
CASE_THEN
. This takes two parameters: the first being a resource to wait for
an event on, and the second a label within the SELECT
block to jump to when
the event occurs. On entry to the SELECT_RES
the task enters a wait state,
when an event becomes available on one of the specified resources control is
transferred to the label specified as the handler. Each handler must end with
either break
or continue
; break
causes control to jump the point
after the SELECT
block, whereas continue
returns to the waiting state to
handle another event.
In the preceding example the timer’s trigger is adjusted into the future each
time it is reached, causing it to trigger indefinitely at regular intervals.
During the handler for the timer event the LED state is toggled only if the
button is currently held. This structure means that the task will spend most of
the time waiting for either of the two possible events. However, the timer event
still occurs even when the button isn’t held - so the handler runs even though
it won’t have any effect. The select construct has the facility to conditionally
mask events which aren’t always of interest - this is called a ‘guard
expression’. In the following listing, the guard expression prevents the
on_timer
event from being handled until its condition is met. The guard
expression is re-evaluated each time the SELECT
enters its wait state; if an
event has occurred whilst ‘masked’ by a guard expression, it will be handled
once its condition is re-evaluated and true.
1#include <platform.h>
2#include <xcore/hwtimer.h>
3#include <xcore/port.h>
4#include <xcore/select.h>
5
6int main(void)
7{
8 port_t button_port = XS1_PORT_4D,
9 led_port = XS1_PORT_4C;
10
11 hwtimer_t timer = hwtimer_alloc();
12 port_enable(button_port);
13 port_enable(led_port);
14
15 int led_state = 0;
16 unsigned long button_state = port_in(button_port);
17 port_set_trigger_in_not_equal(button_port, button_state);
18
19
20 SELECT_RES(
21 CASE_THEN(button_port, on_button_change),
22 CASE_GUARD_THEN(timer, ~button_state & 0x1, on_timer))
23 {
24 on_button_change:
25 button_state = port_in(button_port);
26 port_set_trigger_value(button_port, button_state);
27 continue;
28
29 on_timer:
30 hwtimer_set_trigger_time(timer, hwtimer_get_time(timer) + 20000000);
31 led_state = !led_state;
32 port_out(led_port, led_state);
33 continue;
34 }
35
36 port_disable(led_port);
37 port_disable(button_port);
38 hwtimer_free(timer);
39}
Advanced port I/O#
Ports are extremely flexible and ‘soft-peripheral’ implementations can often delegate significant portions of their work to them; this can drastically improve responsiveness. In particular, ports can be connected to configurable Clock Block resources which can be used to drive timings of port operations. Documenting the full capabilities and interfaces of ports and clock blocks is not within the scope of this document. Further information about these resources can be found in the lib_xcore port and clock APIs, and in the in the XCore instruction set documentation.
Guiding stack size calculation#
As shown earlier in this document, the XMOS tools are able to calculate stack size requirements for many C/C++ functions.
Generally a function’s stack requirement is calculable when its own frame is a fixed size and it only calls functions whose stack sizes are also calculable.
Function pointer groups#
One obvious case where it is not possible to determine a function’s stack requirement is when it makes a call through a function pointer. It is usually impossible to automatically determine all functions which a given pointer may point to. For this reason, the XMOS tools allow indirect call sites to be annotated with a function pointer group. Functions can also be annotated with one or more groups to which they should belong. When the stack size calculator encounters an indirect call through an annotated function pointer, it takes the stack size requirement to be that of the hungriest callee in pointer’s group.
The following snippet declares a function pointer fp
which is a member of
the group named my_functions
:
__attribute__(( fptrgroup("my_functions") )) void(*fp)(void);
When a call through such a pointer appears in a function, that function’s stack
size will include a sufficient allocation for a call to any function which has
been placed in my_functions
. Pointer groups are empty until functions are
added to the explicitly, so such a call is dangerous until all possible callees
have been annotated with the correct group. In the following snippet, func1
is annotated so that it is a member of the group my_functions
:
_attribute__(( fptrgroup("my_functions") ))
void func1(void)
{
}
Explicitly setting stack size#
In some cases it is necessary to explicitly set the stack size allocated for a function. This may be because a function does not have a fixed requirement (e.g. where a variable length array is used) or might be desirable when heavy use of indirect calls makes annotation infeasible.
In these cases it is possible to specify the number of words which should be
allocated for calls to a particular function; the following snippet is an
assembly directive which sets the stack requirement for a function named
task_main
to 1024 words:
.globl task_main.nstackwords
.set task_main.nstackwords,1024
Note that this is simply defining a symbol (whose name is based on the function name) which would be defined by the compiler for a calculable function.
This code can be assembled and linked as a separate object, or can be passed to
xcc
as an additional input file when compiling C/C++ code.
A limited set of binary operators is available for expressing stack size requirements; often this is useful for setting a function’s requirements in terms of functions it calls. The available operators are:
+
and*
$M
- evaluates to the greater of its two operands (useful for finding the hungriest of a list of functions)$A
- evaluates to its left operand rounded up to the next multiple of the right operand (useful for rounding up to the stack alignment requirement)
Parentheses ((
and )
) are also available.
The following directives set task_main
’s stack requirement to 1024 words
plus the greater of the requirements of my_function0
and my_function1
,
all rounded to double word alignment:
.globl task_main.nstackwords
.set task_main.nstackwords, (1024 + (my_function0.nstackwords $M my_function1.nstackwords )) $A 2
When stack sizes are set this way, the compiler will still emit warnings for
unannotated calls through pointers; this can be suppressed with the
-Wno-xcore-fptrgroup
option.
Care should be taken when manually setting stack requirements - a function’s
allocation must be sufficient for its ‘local’ usage as well as all functions it
calls and threads it launches using PAR_JOBS
.