Kevin Day [Sat, 16 Apr 2022 05:01:06 +0000 (00:01 -0500)]
Update: Improve the logic for the read link f_file functions.
The f_file_link_read() and f_file_link_read_at() functions have their dynamic allocation logic improved.
Use link_stat.st_size rather than target->used to pass to readlink() or readlinkat().
Initialize target->used to 0 and then update target->used on success while ensuring the string is always NULL terminated after the target->used.
The unit tests revealed that these can and should be improved.
Kevin Day [Sat, 16 Apr 2022 03:08:40 +0000 (22:08 -0500)]
Update: Add missing checks to f_file_link_hard_at() and use F_file_descriptor.
The F_data_not tests are missing, add them.
The EBADF is returning F_directory_descriptor.
In this specific case, the descriptor is not for a directory per-say but for a general file.
The F_file_descriptor code should be returned in this case.
This has been revealed by the unit tests that I am writing.
Kevin Day [Fri, 15 Apr 2022 21:56:11 +0000 (16:56 -0500)]
Regression: The trailing Unicode sequences don't always show correctly for sequences ending normally.
The commit 85df83a6d846d575657016682c7014a09ac8af4e didn't handle all of the cases it needed to.
I forgot to check the normal behavior and ended up breaking normal behavior while fixing the exception cases.
The width_utf might be -1 for ASCII and performing the subtracting without handling -1 (and then casting it to unsigned) results in bad behavior.
Only handle incomplete character at the end of the stream when the character is actually incomplete.
Kevin Day [Fri, 15 Apr 2022 21:01:40 +0000 (16:01 -0500)]
Update: Directory tests following recent changes.
The change from a7d607fecf1fdd5eff36d53b6108b6cc316bdf1f didn't include test updates.
The tests need to provide properly dynamically allocated structure for the mocks so that the deallocation internal to the called functions work as expected.
Kevin Day [Fri, 15 Apr 2022 20:36:25 +0000 (15:36 -0500)]
Bugfix: Trailing Unicode sequences that get cut off before the encoding is completed improperly display.
Detect when this kind of underflow happens.
Print remaining characters without trying to print missing characters.
Reset and perform remaining data to print at end.
Kevin Day [Fri, 15 Apr 2022 01:10:17 +0000 (20:10 -0500)]
Bugfix: Unicode characters are not printing correctly.
At some point it seems that I accidentally mass converted all of the macro_f_utf_byte_width() and similar to macro_f_utf_char_t_width().
This broke the printing logic.
The f_print is handling normal characters and not f_utf_char_t.
Kevin Day [Thu, 14 Apr 2022 04:04:24 +0000 (23:04 -0500)]
Update: Implement file statistics flag enumeration, dereferencing/no-dereferencing, and also use F_stream_not.
While writing the unit tests for the f_file project I noticed an oversight in the design.
The dereferencing logic needs to be designated rather than assumed.
This requires changing the parameters to functions.
Now is the time to break the API as the official stable release is not made.
This takes the minimalistic approach to changing the API.
I can imagine better designs where I pass the file stat flag everywhere rather than sometimes passing the file stat flag and other times passing a dereference boolean.
This is a larger change in which I wish to avoid.
Improvements to this can be made for the next development series.
The roles can be merged into the file statistics flags.
The roles of group and owner are separated.
For the purposes of limiting changes, both owner and group are treated the same.
The next development cycle will be expected to improve upon this.
The exclusive boolean is now merged into the file statistics flags.
The fake program must be updated.
A new functionality for designating when to dereference and when not to.
The default behavior is to dereference.
This means that the new option is "no_dereference".
To always favor the dereference by default, the file statistics flag is a "reference" flag used to designate that something must not dereference.
The f_file_descriptor() function should return F_stream_not.
This makes more sense as it only fails when the file descriptor is not a valid file stream.
As mentioned above, I was in the process of updating the tests.
This includes some test updates.
Kevin Day [Mon, 11 Apr 2022 02:16:54 +0000 (21:16 -0500)]
Bugfix: Fix problems exposed by f_file project tests.
The f_file_access() should have the mode parameter.
Add missing f_file_access_at().
The f_file_clone() and f_file_copy() are treating exclusive incorrectly.
Do not test for exclusive when assigning the file mode, that is not what exclusive is for.
Add missing paramter check in f_file_descriptor(), f_file_read(), f_file_read_block(), f_file_read_until(), f_file_stream_read(), and f_file_stream_read_until().
The return error result for when a file is not a directory should return F_directory_not rather than F_directory.
Remove unneeded parameter check from f_file_mode_set(), f_file_stream_write_until(), and f_file_stream_write_range().
Add missing range.start > range.stop checks where appropriate.
Many of the file stream functions have locked function calls when unlocked function calls should be used.
Such as where feof() should instead be feof_unlocked() and ferror() should instead be ferror_unlocked().
The private_f_file_stream_write_until() function needs to use unlocked function calls and then must be wrapped between manual flockfile() and funlockfile() calls.
Update documentation comments.
Clean up much of the errno checks alphabetic ordering as appropriate.
Clean up the macro definitions, adding more structure and organization.
Add file access mode define macros.
Simplify function calls where the function may be called with different number of parameters.
This is not strictly allowed by the C language but somehow the POSIX standard got these in and implemented.
This is likely done through macros or some fancy linker tricks.
Avoid these uses and instead use the most likely implemented one.
Example:
openat(at_id, path.string, file->flag);
vs
openat(at_id, path.string, file->flag, mode);
The latter is the one chosen to be most likely.
Add missing result checks that without those checks could potentially alter owner/group on files after an error occurred.
The f_file project tests are a works in progress and so there may be more such fixes to come.
Kevin Day [Sat, 9 Apr 2022 05:42:10 +0000 (00:42 -0500)]
Bugfix: Memory handling related to or exposed by f_directory unit tests.
Replace a double pointer cast with a single pointer cast.
Just in case, if the struct dirent listing is allocated, free it when scandir() returns -1.
The entire struct dirent pointers do not need to be allocated as I thought.
It seems these are copied inside scandir() where the memory deletes inside that function did not affect the mock pointer from the caller.
This resulted in a minor memory leak in the unit test program.
Replace the outermost struct dirent malloc with just a struct dirent array in these cases.
Kevin Day [Sat, 9 Apr 2022 05:25:03 +0000 (00:25 -0500)]
Update: Fix the f_directory unit tests and finish writing them.
The problem with the unit tests not being run (especially on Github) appears to be the result of compiler optimizations.
Fix this by disabling compiler optimizations in the mock functions and add appropriate commetns explaining this.
Kevin Day [Sat, 9 Apr 2022 05:18:25 +0000 (00:18 -0500)]
Bugfix: Invalid memory deletion.
The memory is being incorrectly deleted for multiple reasons.
1) The type is not f_string_t, it is struct dirent.
2) The index within a listing is already a pointer.
3) The listing itself is already a pointer.
4) Remove extra delete commands.
5) The memory is not being deleted on certain errors.
These problems have been exposed by the unit tests.
Also replace macro_f_memory_structure_increment() with f_string_dynamics_increase().
Kevin Day [Sat, 9 Apr 2022 03:46:24 +0000 (22:46 -0500)]
Update: Settings, mock settings, and test settings.
Add standard comments to the settings-mocks and settings-tests files.
Remove the level and monolithic from the settings-mocks and settings-tests files given that I have now determined to rrun the tests exclusively in individual build mode.
Kevin Day [Fri, 8 Apr 2022 03:35:41 +0000 (22:35 -0500)]
Update: Simplify build settings files.
The build setting files contained numerous Objects set to empty content.
This allows for showing how to configure.
With a long standing bug fixed (defaults being applied incorrectly) a misconfiguration as a result of this has been revealed.
Simplify the build settings to make the settings easier to read and maintain.
This ends up fixing the bug (because the default path_sources gets assigned when path_sources is not defined at all).
Kevin Day [Fri, 8 Apr 2022 01:18:52 +0000 (20:18 -0500)]
Update: The fake program should show the fakefile or setting being used when building.
This helps make what is going more clear when viewing the console.
Multiple executions of different build settings (or fakefiles) should be easily determinable such as when running tests.
Kevin Day [Thu, 7 Apr 2022 05:49:09 +0000 (00:49 -0500)]
Update: Improve read program logic to allow for catching interrupts during file read.
Introduce break points in the read process by establishing an outer block loop.
Add the signal check before each block read.
There may need to be additional, smaller checks to further reduce how often the signal check is performed within these loops.
Make sure to use the file stream read rather than the file read.
Calculate the file size and pre-allocate the buffer.
Determine the exact size needed to avoid additional reallocations by utilizing the modulus operator.
The fss_embedded_read is notably out of day and inconsistent with the rest of the fss read programs.
This is only partially updated enough to work with the changes.
Kevin Day [Thu, 7 Apr 2022 05:31:34 +0000 (00:31 -0500)]
Bugfix: For f_file, use off_t, pre-test for EOF or error, use sizeof(f_char_t), and use size_t rather than ssize_t.
The fread() and similar stream functions actually use size_t rather than ssize_t.
Fix this bug by changing ssize_to to size_to for the affected functions.
Use off_t rather than f_array_length_t to be consistent with the types used in the libc/POSIX API.
Add additional checks for feof() and ferror() and the start of the stream read functions.
There needs to be a few more error codes, add a TODO comment to address this via a separate commit.
Use the size of f_char_t rather than 1 for reading.
Use >= in buffer read check rather than ==, just in case.
Kevin Day [Wed, 6 Apr 2022 03:37:30 +0000 (22:37 -0500)]
Update: The example project build settings and fakefiles.
These fell out of sync.
Add support for building via clang.
The "example-objects" project doesn't include support for clang because it is providing a minimalistic example.
Kevin Day [Wed, 6 Apr 2022 03:32:43 +0000 (22:32 -0500)]
Bugfix: Defaults should not be applied when an empty Object is specified.
When an Object is not specified, such as with "path_sources", then the default should be loaded.
When an Object is specified but ha Content with a length of 0, then the Content should not have a default applied.
The example bzip fakefile exposed this problem where the "path_sources" is empty and should be.
The existing code is incorrectly applying the default when length is 0.
The code to actually check for whether or not an Object is specified is already in place but it is not used.
Update the code to utilize the existing information.
Kevin Day [Tue, 5 Apr 2022 03:35:54 +0000 (22:35 -0500)]
Bugfix: The f_capability should accept f_string_constant_t rather than f_string_static_t.
The name.string is incorrect and should be name->string.
Once I fixed this, I noticed a compiler warning about losing the "const" frrom the return value of cap_mode_name().
Use f_string_constant_t rather than f_string_static_t.
Kevin Day [Sun, 3 Apr 2022 15:59:05 +0000 (10:59 -0500)]
Update: Add f_directory tests then fix problems discovered problems and perform appropriate cleanups.
The "preserve_path_headers" in settings-mocks and settings-tests should be set to "yes".
Add missing dependencies-tests file for any project providing tests.
Remove at_id <= 0 parameter checks because this is already handled by EBADF.
Fix alphabetic ordering of errno returns for f_directory.
I started doing this for f_file as well until I realized how much there is to do.
I won't continue this for f_file because I don't plan on adding tests for f_file for this next release.
The ENOTDIR should return F_directory_not rather than F_directory.
Unable to perform many of the tests because the dynamic linker wrapper is not properlly handling stat() and scandir() as expected.
These tests are commented out, incomplete, and comments are added describing why.
The commit that fixed the reversed endianness problems is 78daa0d760fbcc767dd099d68ae02d23b3df4991.
These changes revealed the test problems due to the tests now failing.
Fix the tests.
Make a new character type for the EOL, EOS, and placeholder in a way that is more consistent with the character macros for the f_char_t.
The F_utf_substitute_open_box_d_length and F_utf_substitute_open_box_shouldered_d_length should have _s_ rather than _d_ in their names.
Change the UTF-8 seek to functions to not need to check for URL fragments.
Change the UTF-8 seek to functions to accept f_utf_char_t rather than f_char_t.
This completely makes more sense.
This allows for searching a f_utf_string_t using a f_utf_char_t.
The tests are expecting this behavior earlier and they only happened to succeed due to the coincidence of the bitwise order of the previously incorrect tests.
This swaps the bitwise order of the strings.
More than likely I need a macro check to handle big endian as well.
This is left to changes in the future.
The 0.6.x stable series does not guarantee big endian support.
The 0.8.x stable series does guarantee big endian support because I will be working on adding networking support.
Leave the bitwise improvements and tests for the next development cycle so that I can focus on pushing out the 0.6.x stable release.
If I have the time, I might still fix this after the release candidate is released as I will most likely be improving the tests and fixing any discovered problems.
Kevin Day [Sat, 2 Apr 2022 22:56:05 +0000 (17:56 -0500)]
Update: Get the controller program in a better state for a stable release.
I started realizing that there were things that I forgot all about that the control and controller programs need to do.
There are too many changes to make immediately before the release.
Remove the incomplete code to avoid having stale code and therefore stale binary data.
Comment out the parts that generate socket related information as the socket cannot be supported yet.
Kevin Day [Sat, 2 Apr 2022 22:48:44 +0000 (17:48 -0500)]
Update: Get the control program in a better state and update the control/controller documentation.
I started realizing that there were things that I forgot all about that the control and controller programs need to do.
This is too many changes to make immediately before the release.
Wrap up the control program into a tolerable state and it will be published as is.
Release notes will be used to document that this program didn't make the cut in time.
Kevin Day [Sat, 2 Apr 2022 14:03:20 +0000 (09:03 -0500)]
Bugfix: Incorrectly checking byte width when already inside a character.
The width has already been determined, so these values are parts of a UTF-8 character.
The code is doing a byte check and attempting to verify or process this as a new UTF-8 character sequence.
This results in an inaccurate data from that point forward for the rest of the file.
Kevin Day [Sat, 2 Apr 2022 13:41:07 +0000 (08:41 -0500)]
Update: Do not use sizeof(char_t) for explicitly handling 1 byte.
The char_t could potentially be of some size other than 1.
This read is per-byte and not per-character.
Normally these or true, but some special architectures this might not be.
Kevin Day [Sat, 2 Apr 2022 13:16:39 +0000 (08:16 -0500)]
Regression: Accidentally reversed the big vs little endian bits.
I had a fear that I might of done this with the commit dda60cc8d9cad69b55e7d79b4c052677b03b4413.
Turns out I was right.
All that back and forth getting confused before I realized that I should not use the BIG_ENDIAN macro did result in me getting confused on what bit goes where.
Kevin Day [Sat, 2 Apr 2022 04:08:41 +0000 (23:08 -0500)]
Update: Centralize common error function, minor cleanups, and improve status code programs.
Printing an error for two different long parameters that cannot be used together is very common.
Move this into a single function inside of fll_program.
Make sure that these calls are wrapped in appropriate verbosity checks.
The status code programs is now able to print the number with the error bit and/or warning bit set.
The status code programs status codes error and warning can be used together when using the -n/--number.
Perform minor cleanup where the "main->parameters.remaining.used > 0" can be simplified.
Kevin Day [Fri, 1 Apr 2022 04:48:29 +0000 (23:48 -0500)]
Update: Remove inefficient and pointless size increase.
This must have been by accident or by habit.
The (range->stop - range->start) for these shouldn't be happening.
These checks are solely for ensuring there is enough room after what is already allocated.
The behavior is actually potentially doubling its size.
Only increase size by the required amount.
There are likely other such problems that need to be fixed across this project.
I will likely not get to these by the 0.6.0 stable release and will have to address them following that as I find them.
Kevin Day [Fri, 1 Apr 2022 04:42:01 +0000 (23:42 -0500)]
Update: Ensure all programs are handling signals, adding signal_received to fll_program_data_t.
Make sure fll_program_data_t supports storing the signal.
Move the commonly written functions that are pretty much identical across programs into fll_program:
- fll_program_standard_signal_received()
- fll_program_standard_signal_state()
Increase standard signal check from 10000 to 20000.
I really dislike the signal_check logic that exists purely as a work-around to the signal handling.
Most likely the only way to do this nicely that is available without rewritting the kernel and the libc will be to implement threads.
This will not be done for the 0.6.x versions and will likely happen for the 0.8.x versions.
Kevin Day [Thu, 31 Mar 2022 01:31:52 +0000 (20:31 -0500)]
Update: Add umask integer to fll_program_data_t.
This should be a standard practice that ideally must go away if the POSIX standard ever fixes that idiotic umask() design.
Never under any circumstances should a read-only operation result require a write operation!
I am forced to work around this, so I might as well provide a standard practice work-around.
Kevin Day [Thu, 31 Mar 2022 00:48:03 +0000 (19:48 -0500)]
Update: Add child integer to fll_program_data_t.
A few of the programs practice returning the child return status or the child process id.
Make this behavio standard by adding child to the fll_program_data_t.
Kevin Day [Thu, 31 Mar 2022 00:06:18 +0000 (19:06 -0500)]
Security: Invalid write in FSS processing functions due to improper allocation size.
The start and stop ranges are inclusive.
This means that the size is (stop - start) + 1.
The problems happens where the code is adding additional digits to represent end of line or other special characters.
When this is added, I seem to have forgotten to add the additional numbers to the + 1 and instead replaced the + 1.
This results in the size being potentially short by a single character and thus an invalid write is possible.
Kevin Day [Wed, 30 Mar 2022 23:47:13 +0000 (18:47 -0500)]
Update: Move the program data into the main for fss_embedded_list_read.
Replace fss_embedded_list_read_main_t with fll_program_data_t and fss_embedded_list_read_data_t.
Move parameter initialization and deallocation for FLL program data into main.c.
Kevin Day [Wed, 30 Mar 2022 03:37:07 +0000 (22:37 -0500)]
Bugfix: The out of range is printing and then continues processing.
When out of range happens an error should be printed and should stop processing for that parameter.
This is not happening because the status is incorrectly being returned when status is not assigned an error bit.
The out of range check happens independent of the status variable.
Kevin Day [Wed, 30 Mar 2022 01:17:09 +0000 (20:17 -0500)]
Update: Use signal_check more consistently in programs.
There are programs that do not use the fll_program_data_t structure.
These now define the signal_check in their main.
Such cases will go away as soon as they are converted into using fll_program_data_t.
Kevin Day [Tue, 29 Mar 2022 03:38:43 +0000 (22:38 -0500)]
Update: Finish writing tests, fix endianness, simplify some code.
Now that f_char_t exists and is in use, rename f_utf_character_t to f_utf_char_t to be consistent (and simpler).
The endianness detection is incorrect.
The BIG_ENDIAN is not represented as expected.
Change the detection to use GCC's __BYTE_ORDER__ macros.
Fallback to little endian if those macros are not available.
It took a while to figure out this was the problem and I got rather confused going back and forth between byte orders.
I think the changes I made are correct, but I need additional better testing on the byte orders.
Kevin Day [Mon, 28 Mar 2022 04:30:30 +0000 (23:30 -0500)]
Update: Bring f_utf project in sync with f_string.
I have been explicitly avoiding making change to f_utf until I finished updating and successfully vetted the f_string project.
Now that the f_string has tests that pass, mass update the f_utf project.
This changes the functions and structure to be inline with how f_string works.
Many of the fixes are brought over.
I have observed that several of the bugs that were recently in f_string is also present in f_utf.
These bugs in f_utf should now be fixed as well.
This tests are brought over.
Only the tests for functions that are shared between f_string and f_utf are implemented.
And then those tests currently cannot be compiled.
The reason is that normal quoted strings cannot be used with the special UTF dynamic strings.
I simply did not have the time to get to this to finish writing these tests for f_utf.
There are some parts of f_string that make no sense in f_utf, such as f_string_range_t and f_string_quantity_t (both of which can be used on f_string and f_utf strings).
These have been omitted when copying the relevant f_string into f_utf code.
Kevin Day [Mon, 28 Mar 2022 00:56:35 +0000 (19:56 -0500)]
Update: Restructure parts of f_utf project, and use f_utf_t, and add 'u' to hexdigits in common.h.
Move the relevant functions into utf/convert.h, utf/is.h, and utf/is_character.h.
Implement f_utf_t as a type of uint32_t.
This should allow more customizability on that type if need be.
It also provides a more explicit context.
Add 'u' to designate the hexdigits are unsigned (This likely more specifically means it is an unsigned int).
This might be removed if it becomes a problem with architectures where "u" represents 16-bits or less.
Due to the size of the work, I am avoiding making this change in all of the Unicode comparison functions.
There are quite a lot of hexdigits in use.
Kevin Day [Sun, 27 Mar 2022 22:22:18 +0000 (17:22 -0500)]
Bugfix: Partial string functions are incorrect.
The tests exposed these problems.
The *_assure() functions are operating as if they are *_assure_nulless().
Remove the NULL checks that should not be there and fix the documentation comments.
The *_assure() functions are not considering the following:
1) The case where range.stop is >= source.used.
2) The fact that range.stop is inclusive and should not be directly used in the same way that *.used is used.
In the case of (1), if the stop range result in it overflowing past the actual length, the previous code results in invalid reads.
In the case of (2), the stop.range needs to instead be stop.range + 1.
Kevin Day [Sun, 27 Mar 2022 16:00:28 +0000 (11:00 -0500)]
Update: Add additional tests for f_string project.
This adds tests for the f_string functions for functions defined in string.h.
This adds tests for the f_string functions for functions defined in dynamic.h that functionally mirror functions defined in string.h.
There are still more functions in dynamic.h that need to have tests written for them.
Kevin Day [Sun, 27 Mar 2022 15:19:04 +0000 (10:19 -0500)]
Bugfix: Incorrectly performing nulless string appends and prepends.
These problems are exposed by the tests that I am writing.
The destination->used < length check cannot be performed because of the presence of NULL characters.
The nulless versions of the strings may be identical even if their sizes do not match.
The only case where this check should still happen is when destination.used is 0.
The comparison operator is using the wrong variable in some cases ('j' should be used with 'destination').
Kevin Day [Sun, 27 Mar 2022 01:41:37 +0000 (20:41 -0500)]
Bugfix: Print functions are not always printing the entire string.
I observed this when working with printing binary data in '%Q'.
The fwrite_unlocked() might prematurely return but not on an error.
If it does, first check to see if it is an error and then keep trying until everything is printed.
This is more of a band-aid fix.
This requires an additional variable and an additional loop.
I would rather avoid all of this and so a re-design is likely necessary for greater efficiency.
As this is not my focus at the time I am going to tolerate this less than desirable design.
Kevin Day [Sat, 26 Mar 2022 16:22:39 +0000 (11:22 -0500)]
Bugfix: Increasing strings by too much.
The *_increase_by() methods already include the ".used" length.
The previous code is also adding the ".used" length resulting in an increase of ".used" * 2.