Complete C standard library
Posté le 19/08/2018 19:31
Motivation
Until now there was no complete C standard library (aka
libc) available for the Casio calculators. Although some parts of this library have been provided by fxlib and
gint, there was no libc implementation complying with the standard and compatible with the sh3eb architecture ready to use.
To change that, I decided to port
newlib to the Casio CPU. Newlib is an alternative libc implementation intended for use on embedded systems.
Alpha
Follow this link and click the download button in the top right corner:
>>> v1.1 <<<
Instructions on how to install newlib alongside with gcc (big shout-out to Lephé):
Compiler sous Linux avec GCC
Features for Casio fx9860g calculators:
* C standard library
libc
→
printf implementation to print text to the display
→ Dynamic allocation of memory using
malloc and
free
→ Memory manipulation using
memcpy,
memcmp,
memset etc.
→ String manipulation using
strcpy,
strcmp,
strstr,
strtok
→ ...
* Math library
libm
→ Floating point arithmetics
→ ...
* Automatic library and include path recognition after installation
* Basic Casio features:
→ implementation of
GetKey,
Bdisp_AllClr_DDVRAM,
Bdisp_PutDisp_DD,
Print and
locate without fxlib (but you can use it if you want)
Code
To contribute or get all those bleeding edge features, see the code including all further information:
libc (my GitLab repository)
The project you find in my repository is a fork of
the official newlib repository. To make it easier for everyone to follow, I try to keep a clean
git history. That means that all my changes are located on a dedicated branch with meaningful commits.
I also try to keep the changes to the upstream library minimal. That increases maintainability a lot.
Contributing
If you have a ideas, feature request, found a bug or simply want to contribute, just send me a message and you're in! You can also create
Issues and
Merge Requests directly in the repository. As in every OpenSource project: merge requests welcome!
Citer : Posté le 22/08/2018 20:11 | #
Today I managed to setup some syscalls listed here. That means for now I'm testing newlib without any library to avoid conflicts. Later I'll add the support for fxlibc/gint.
Do you think it makes sense to replace the libc implementations of free, malloc, memcmp, memcpy, memset, strcat, strcmp, strlen, strncat, strncmp, strncpy, strrchr etc. with syscalls?
Ajouté le 22/08/2018 à 20:15 :
This implies that I call the addresses directly, gint will probably not have any control over it...
Citer : Posté le 23/08/2018 08:51 | #
Seems fair. Not that there is any major difference!
Edit: I should add that fxlib is just a bunch of syscalls so you're already supporting it in some way. ;D
I think the system heap must remain accessible. We can rename the syscalls to sys_malloc() and sys_free(). Please bear in mind that:
- On monochrome SH3, there is no spare RAM where newlib's heap can be installed (?)
- On monochrome SH4, there is a 256k free buffer but developers won't use it freely because of SH3 compatibility
- On Graph 90+E, the RAM layout is not fully known but there is little to no spare space at all.
As for the "pure" str- and mem- functions, it's worth trying out newlib's version. It could turn out to be a lot faster since CASIO's operating system has rarely demonstrated high optimization levels.
This is not a problem as long as you don't use syscalls that manipulate hardware while gint is active. All the functions you mentioned above are fine, in fact even drawing and pushing to screen with Bdisp_PutDisp_DD() are fine. (Note: On fx-CG 50 there are some reentrancy issues, but let's keep it simple).
Citer : Posté le 23/08/2018 10:47 | #
I just recently dived into the world of Casio, that means my knowledge of the memory structure of the Casio is very scarce.
On monochrome SH3, there is no spare RAM where newlib's heap can be installed (?)
Ok so let's focus on SH3 for now. There is no spare RAM? No RAM -> no heap -> no malloc? I think even if there is very little RAM, we should implement malloc (or am I misunderstanding you?)
To make malloc work, I need to implement _sbrk which allocates more heap. My problem is that I do currently not really know how xD
Here is what I (believe to) know: with the fxlib linker script there are the 4 output sections .text, .rodata, .bss and .data. During runtime, .bss is naturally in RAM while .data will have to be loaded to RAM by crt0.s
Now I don't know where is stack and where is heap. That makes a bit hard to implement _sbrk which is supposed to allocate more heap. I also do not really know what the system does (as you mentioned, there's probably a heap already used by the system).
Citer : Posté le 23/08/2018 11:01 | #
You can implement malloc(), but you need to give it a chunk of RAM to make it work. The system has its default 48k heap, but there's pretty much nothing else. The stack is 16k and the static, mapped RAM is 8k.
gint uses another RAM chunk on monochrome machines, which is probably not intended to be used by add-ins. It just appears to be unused by the system. On SH4 part of it at least is used so I had to shrink the region. You could use this region, which is around 8k, although that would definitely break gint compatibility. (But if it's only malloc() then that's fine.)
On a modern computer system sbrk() requests heap pages to the kernel. Here we don't have any dynamic paging system so you'd have to expand your heap chunk in a fixed way.
For instance, you start with 2k at 8800'e000 (gint's region) and when sbrk is called, you extend towards higher addresses until you reach the maximum size of 8k. Or you can just start with the full 8k and deny all sbrk requests. I think this is what makes the most sense.
That's right.
Ok, so the location of the stack and heap is OS-version-dependent. All that we need to know is that the stack pointer r15 is set a suitable value when the add-in starts, and that syscalls malloc() and free() operate with the system heap.
Don't be mistaken regarding the system heap: when we say sbrk has to allocate more heap, it means more of your heap, not the system's. If you wish to use newlib's malloc(), I reckon you need to do it independently from the system, so you have to manage you own piece of RAM. The location of the system heap should be "irrelevant".
Citer : Posté le 23/08/2018 11:43 | #
I think with system calls, my newlib is at a first usable state. Before committing I want to discuss this malloc issue a bit more xD.
I see that if I implement malloc on my own, I need some free RAM (which should not collide with gint and be as compatible as possible). I guess we could build in some configure switched or similar to ensure compatibility...
I think the system heap must remain accessible
To be honest, I still don't see where's the problem with using the 48k system heap. What will be the problem if I make it accessible to the developer via malloc? Currently I see no alternative since imo there should be at least 16k heap if possible.
Citer : Posté le 23/08/2018 12:02 | #
Configure switch seems a good compromise between compatibility and features.
The problem is that newlib's malloc() will start writing bytes in the area. As you probably know the heap is usually a kind of linked data structure such as a list or a binary tree. It goes without saying that newlib's data structure differs from the system's. Basically if you write any data in here you will break the system heap, which is used by virtually all add-ins and default applications.
Citer : Posté le 23/08/2018 12:09 | #
Couldn't you just take the malloc() from fxlib, if you're going to reimplement it anyway (so it's not newlib) ?
Ecrivez vos programmes basic sur PC avec BIDE
Citer : Posté le 23/08/2018 12:20 | #
Ooh I understand! I think we are not talking about the same thing. My plan was to overwrite the newlib's malloc by simply calling the Casio syscall of malloc. This will use the system's heap just like when I call malloc in fxlib. The system heap will not care who uses its syscall and what data will be written. Am I right?
@Zezombye That's that I meant.
For the configure switch: actually I prefer #ifdef switches since they can be changed at application compile-time (rather than at libc compile-time).
Citer : Posté le 23/08/2018 12:43 | #
Ooh I understand! I think we are not talking about the same thing. My plan was to overwrite the newlib's malloc by simply calling the Casio syscall of malloc. This will use the system's heap just like when I call malloc in fxlib. The system heap will not care who uses its syscall and what data will be written. Am I right?
Wow that makes much more sense indeed. Yes this would the trivial implementation. That would totally work.
How do you plan to do it on a practical level? I guess you could exclude newlib's malloc-related files from the archiving and a custom object file? Not sure what the most maintainable solution is.
I had not considered this option, now that you mention it I can only agree. If I'm not wrong, this is a switch to choose between the "syscall" and "separate heap" implementations. So, you'd have to compile both versions (under different names) when building newlib and then use the preprocessor switch to change the definition of malloc() from the headers, is that right?
Citer : Posté le 23/08/2018 13:10 | #
How do you plan to do it on a practical level?
First I expected the malloc definition to be weak, but I found no such attributes. However, I tried to just define it anyway and it worked out of the box. Apparently the target-specific definitions take precedence xD. No ugly deleting or modification of existing sources needed.
So, you'd have to compile both versions (under different names) when building newlib and then use the preprocessor switch to change the definition of malloc() from the headers, is that right?
In a source file, the preprecessor can decide where to look for headers. Unfortunately it does not work the other way round: the preprocessor cannot decide where the linker looks for definitions. (And even if it could, all definition are bundled into the library at newlib's compile-time anyway.)
Hmm, I think in this particular case it's not possible to switch definitions. The #ifdef directive can only reside in the headers, because the sources can't change after newlib's compilation. However, the header of malloc cannot contain definitions... :/
I didn't really think about this, so maybe I overlooked something.
Citer : Posté le 23/08/2018 13:43 | #
So you defined it in /newlib/libc/sys/sh3eb and built from that?
What you said is right in principle, but can be worked around in practice:
#define malloc malloc_newlib
#else
#define malloc malloc_sys
#endif
I didn't really think about this, so maybe I overlooked something.
Switching names is really the only idea I have. It has some drawbacks (won't work if you forget the header, unless one of the two implementations is used as the "default", needs preprocessor in assembler files) but I think it's worth doing it because of the modularity it brings to user code.
Citer : Posté le 23/08/2018 14:00 | #
So you defined it in /newlib/libc/sys/sh3eb and built from that?
Yes exactly, i uploaded the new version. Note that if you include sys/casio_syscalls.h, basic Casio functions will be functions will be available for testing:
int GetKey(unsigned int* keycode);
void Bdisp_AllClr_DDVRAM(void);
void Bdisp_PutDisp_DD(void);
void Print(const unsigned char* str);
void locate(int x, int y);
void wait_ms(unsigned int ms); // TODO signature assumed
What you said is right in principle, but can be worked around in practice:
Oh right that should work provided that the #define is processed before any other header that uses malloc. That means I probably have to put it into stdlib.h... I'll see. At the moment there is only one implementation anyway xD
Should I even bother providing an alternative implementation to the Casio syscall? Would it help for gint? At the moment I don't really see the benefit so I guess I'll postpone implementing malloc_newlib and spend some time on testing.
I'd really appreciate if you'd like to do some testing, too. If you find issues, just tell me or write a GitLab issue. Markdown is just much better there and it's easier to link code (or anything really). Btw: I already wrote a little issue to you a few days ago, maybe you didn't notice :P
Citer : Posté le 23/08/2018 14:12 | #
This is not a trivial question. On fx9860g the heap is 48k and is easily the largest memory area available apart from the half-RAM 256k region that is only available on SH4, so it's pretty useful already.
On fxcg50, the user has to work with a 128k heap and a 512k shared data/stack region with the stack growing from the end. A user that wants to enlarge the heap may want to use another implementation of malloc() with a customizable size of heap.
However the next fxcg50 update will introduce Python programming, and Python cannot run on 128k. I have tested the new Python models during a CASIO event in France and it turned out that the heap was extended to 3M! If the OS update from August, 31 (which introduces Python) indeed provides such a large piece of memory, then I'd consider a newlib_malloc() essentially useless. I suggest to wait until then before trying to implement it.
Completely missed it, sorry. The community here doesn't have the habit of using issue trackers, although we tried. I'll look into it. Ok for the tests.
Citer : Posté le 23/08/2018 17:51 | #
Ok so for this machine, I'll stick to the Casio syscall malloc for now.
However the next fxcg50 update will introduce Python programming
Neat! I'm looking forward to that!
Ajouté le 23/08/2018 à 17:54 :
### UPDATE ####
I proudly present the early alpha release v1.00.
It also supports printf now, have fun!
Citer : Posté le 23/08/2018 17:59 | #
Wow. GG for having done this so fast
Citer : Posté le 23/08/2018 18:05 | #
As a side note the KhiCAS add-in for symbolic computation is driven by a Python-compatible language that's much more powerful than the incoming update, which only has 2 basic modules.
I was wondering how you'd implement printf(). I suppose you used Print() and relied on locate() for the positioning?
Citer : Posté le 23/08/2018 18:06 | #
Thank you, but don't praise me too fast
There is still a lot of work to do. Now some testing has to be done, writing instructions, fixing bugs, adding further syscall features, establishing compatibility to fxlib and gint, ... xD
Ajouté le 23/08/2018 à 18:15 :
Ok to be fair it's a very basic implementation: there is a syscall write which is called by printf. I simply put a Print in there. Since I don't want to lose flexibility here, the user has to call locate first. There is also no bounds-checking implemented yet. I plan to add this in the future.
However I fear that if you include fxlib, I might lose control over locate and might not be able to check bounds anymore, but we'll see about that one.
To be honest, although that is definitely not scope of libc, I consider adding as many Casio syscalls as possible to render fxlib useless.
What do you think?
Citer : Posté le 23/08/2018 21:12 | #
The user will always have the possibility to invoke the locate() syscall themselves. You can only check bounds if you know the address of the cursor position in memory. It's a trivial matter to find this address by disassembling the syscall. However it's OS-version-dependent and we don't have a serious list of addresses for each version. In the end I'd say you can't do things like automatic newlines.
Are you sure you don't want to include them in a separate library? For the end user it wouldn't be too much work to add -lsyscalls and it would keep things cleanly separate. My instinctive opinion about this is that libc.a should try to expose only standard symbols
I'm saying this because as a gint user, if I call a hardware-related syscall, the odds are I'm going to blow up my calculator. There are a few, scarce difficulties when integrating the libc into gint (such as where to print when using printf()); I think these can be solved by thinking about them for long enough, but for syscalls, I cannot do much. This is why I suggest putting them in a different archive.
Now packaging the syscalls in a library is something that will help and I think it's a good idea!
As a side node, here's what I mean about difficulties with printf(): gint does not have something as fancy as locate(). Currently it has the int dprint(int x, int y, const char *fmt, ...) call which does exactly what you think it does, but it cannot support printf() because it lacks a global notion of position. I'm pondering writing a console library that would implement printf() in a natural way; another possibility is to unify file I/O with external communication (serial, USB) and develop a real write() call. I'm probably going to do that since I consider gint's role to define such a function (as I see it as kernel).
Citer : Posté le 23/08/2018 21:33 | #
The user will always have the possibility to invoke the locate() syscall themselves. You can only check bounds if you know the address of the cursor position in memory
While I agree that I cannot prevent the user to call the sycalls directly and break my system, I could simply permit it xD
However...
My instinctive opinion about this is that libc.a should try to expose only standard symbols
My instincts tell me the exact same thing
Now packaging the syscalls in a library is something that will help and I think it's a good idea!
Now that's something that has already been done by someone here, right?
I'm pondering writing a console library that would implement printf() in a natural way; another possibility is to unify file I/O with external communication (serial, USB) and develop a real write() call. I'm probably going to do that since I consider gint's role to define such a function (as I see it as kernel).
First of all, you should not implement printf, but the backend of write (which is called when something is written to the terminal).
Secondly, you can have write to both the terminal and serial, USB, all together. write takes a file descriptor (If I recall correctly, 0, 1 is stdio and 2 is stderr). Just add additional streams et voilà
Citer : Posté le 23/08/2018 21:46 | #
Not that I know. Simon Lothar has listed all the (known) syscalls, and there has been some libraries using them, but a good complete, packaged, documented thing, I fail to see.
Right, this is the kernel way of doing things. Thanks for the reminder.
I find the way you elude how hellish USB is, fascinating.
Citer : Posté le 23/08/2018 22:04 | #
[...] a good complete, packaged, documented thing, I fail to see.
I'm surprised that there are so many cool applications here and yet not a really solid basis (hardware support, libc etc.). (So good job creating gint xD)
I can't deny dprint() or sprintf() to the non-libc user since I have a lightweight implementation at hand.
Of course not, classical case for #ifndef __NEWLIB_H__
I find the way you elude how hellish USB is, fascinating.
To be honest I never developed apps for USB. Right after I started all those cool ideas that sound much better in my head than they are in reality... and finished them about 60% just to abandon them - I might collaborate with you implementing USB to... (which device do you even want to communicate to? You'd have to write a linux usb driver... That requires a good use case for motivation)