[GCC-XML] GCC backend upgrade

[Alex Leach]

Dear Brad / list,

Long story short, I thought I'd have a go at upgrading GCCXML's backend
compiler to GCC4.7.2. I made a fork on github a week ago and have been
working on it since.

I initially thought that there was no chance of me getting it to work, but
I stuck at it and have had some joy, sort of. Parse-able XML output is
being dumped, but there are some errors reported by the backend compiler.
gccxml_cc1plus exits with code 1, and gccxml assumes that it seg-faulted,
so all the ctests currently fail..

The last error I've come across has stumped me.. When looking for a
solution online, I found a thread in this mailing list
(http://public.kitware.com/pipermail/gccxml/2008-April/001100.html), where
you resolved the issue within 24 hours - Kudos for that Brad!

On github, I found the commit to workaround this:
https://github.com/gccxml/gccxml/commit/36c88f006a8e81261e637869b985332308ac99dd

But tbh, I don't have a clue how to implement a similar fix for GCC 4.7.2.

The tracebacks only seem to show include lines and errors within system
headers, which is new to me and makes debugging it a problem...

For example:

In file included from
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.2/../../../../include/c++/4.7.2/ios:43:0,
from
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.2/../../../../include/c++/4.7.2/ostream:40,
from
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.2/../../../../include/c++/4.7.2/iostream:40,
from ../GCC_XML/GXFront/gxSystemTools.h:29,
from ../GCC_XML/GXFront/gxConfiguration.h:20,
from ../GCC_XML/Testing/TestUsualInclude.cxx:1:
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.2/../../../../include/c++/4.7.2/bits/ios_base.h:791:5:
error: 'std::ios_base& std::ios_base::operator=(const std::ios_base&)' is
private
In file included from
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.2/../../../../include/c++/4.7.2/ios:45:0,
from
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.2/../../../../include/c++/4.7.2/ostream:40,
from
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.2/../../../../include/c++/4.7.2/iostream:40,
from ../GCC_XML/GXFront/gxSystemTools.h:29,
from ../GCC_XML/GXFront/gxConfiguration.h:20,
from ../GCC_XML/Testing/TestUsualInclude.cxx:1:
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.2/../../../../include/c++/4.7.2/bits/basic_ios.h:64:11:
error: within this context
In file included from
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.2/../../../../include/c++/4.7.2/iostream:41:0,
from ../GCC_XML/GXFront/gxSystemTools.h:29,
from ../GCC_XML/GXFront/gxConfiguration.h:20,
from ../GCC_XML/Testing/TestUsualInclude.cxx:1:
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.2/../../../../include/c++/4.7.2/istream:
In member function 'std::basic_istream<char>&
std::basic_istream<char>::operator=(const std::basic_istream<char>&)':
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.2/../../../../include/c++/4.7.2/istream:56:11:
note: synthesized method 'std::basic_ios<char>&
std::basic_ios<char>::operator=(const std::basic_ios<char>&)' first
required here


--------------------------------


There are similar errors with std::basic_ofstream, std::basic_streambuf,
std::locale::facet, std::num_put, and std::ctype. All giving errors about
copy constructors on private members.

I suspect this has to do with a change in the C++03 standard, judging by
this answer on SO: http://stackoverflow.com/a/9123195/566098

If you could suggest on which files I should direct my efforts, I'll do my
best to resolve this. Any help would be really appreciated! Otherwise,
I'll spend some time grep'ing and preprocessing various source files.. Any
ideas?

--------------------------------

Changes I made are in a fork of your repo on github, hosted at
https://github.com/alexleach/gccxml/

The changes are split over a few commits, as changing the GCC code
affected almost 3,000 files, so I dumped that in a single commit and any
manual edits and patches were committed afterwards. Should have probably
started a branch...

Most of the work involved editing the CMake build system files. I saw that
you're a core CMake maintainer, so I apologise if my edits aren't up to
your usual standards!

---------------------------------

Unrelated to these issues, but thinking about moving forward.. Richard
Guenther - one of the lead maintainers of the GCC project - suggested to
me on IRC the other day that I might try and turn it into a plugin...

Plugins seem to be compatible with GCC versions 4.5 and up. I thought it
sounds like a good idea, in that it would hopefully drop the need for a
lot of the support headers; I imagine it should reduce future maintenance;
the binary is much smaller; and obviously compiling a dynamically loaded
plugin is much, much quicker than recompiling a large chunk of GCC.

So after getting the above issues - which I think are totally resolvable -
I started working on patching xml.c to become a plugin. Obviously there's
some issues with the global variables and functions patched into other
modules, like in method.c, decl2.c and diagnostic.c, but the hard part is
already done; you did it years ago!

I should probably create a fork for the plugin, but so far, I've just
created a new sub-project (GCC/gccxml-plugin) in which I'm hacking xml.c
in an attempt to turn it into a plugin. Still quite a lot of work to do;
I'm a noob with C and C++ so progress is slow, but I'm learning a lot!
Would also need to modify the gccxml driver to instead invoke g++, also
passing it different arguments; i.e: '-fplugin=libgccxml_plugin.so
-fplugin-libgccxml_plugin-xml=... -fplugin-libgccxml_plugin-xml_start=...'


---------

Thanks for reading this! I was indenting on submitting a Pull Request at
some point, but I feel like I've made a mess of what was beautiful code
and a clean version history, so maybe it's best left as a separate fork..

If you would be interested in merging any of it back into the main
repository, please let me know and I'll make a Pull Request. I think I
should get it working first, though...

One last thing worth mentioning. I had a look at the other fork, by Andrej
Mitrovic, and incorporated some of his feature additions, like putting C++
attributes into XML tags and detection of Pod, Anon and TypeScoped C++
Types.

Yours sincerely,
Alex

[Alex Leach]

Forwarding back on to list..

------- Forwarded message -------
From: "Alex Leach" <beamesleach at gmail.com>
To: "Andrej Mitrovic" <andrej.mitrovich at gmail.com>
Cc:
Subject: Re: [GCC-XML] GCC backend upgrade
Date: Sun, 10 Feb 2013 20:24:01 -0000

On Sun, 10 Feb 2013 19:42:02 -0000, Andrej Mitrovic

Post by Alex Leach
I started working on patching xml.c to become a plugin.

Hi. Is this hosted somewhere? I'd sure like to take a look.

Hi Andrej!

Thanks for the reply! Sorry, haven't got around to pushing that yet...
Trying to keep my commits clean. Will do it now and will forward a commit
link.. Be warned, though, I only started on it yesterday (and it is the
weekend)!

A couple reference places I've been using, other than the GCC Internals
manual:

1. Part 1 and 2 of this blog post:
http://www.codesynthesis.com/~boris/blog/2010/05/10/parsing-cxx-with-gcc-plugin-part-2/
2. gcc-python plugin source code:
http://git.fedorahosted.org/cgit/gcc-python-plugin.git/tree/gcc-python-version.c

Kind regards,
Alex

[Alex Leach]

On Sun, 10 Feb 2013 20:48:57 -0000, Andrej Mitrovic

Post by Alex Leach
http://www.codesynthesis.com/~boris/blog/2010/05/10/parsing-cxx-with-gcc-plugin-part-2/

Hmm is this Posix only? (i.e. only SO support but not DLLs?)

Does it say that? Only just started that second post.. I've configured the
CMake build system to use a system-agnostic file name suffix; there's some
variable like ${CMAKE_SHARED_OBJECT_SUFFIX} which I used. I thought it
should work on any platform with GCC installed and a dynamic loader. Does
that not include DLLs on Windows? I can ask on #gcc's irc channel if
necessary?

Just pushed a 'plugin' branch to my github repo. Try:

git clone git at github.com:alexleach/gccxml.git
git checkout plugin

Cheers,
Alex

[Alex Leach]

On Sun, 10 Feb 2013 20:58:46 -0000, Alex Leach <beamesleach at gmail.com>
Direct link:

https://github.com/alexleach/gccxml/tree/plugin/GCC/gccxml-plugin

[Alex Leach]

On Sun, 10 Feb 2013 20:48:57 -0000, Andrej Mitrovic

Post by Alex Leach
http://www.codesynthesis.com/~boris/blog/2010/05/10/parsing-cxx-with-gcc-plugin-part-2/

Hmm is this Posix only? (i.e. only SO support but not DLLs?)

I've just been reading up a bit on this on the GCC mailing list.

It seems that the GNU hello plugin has been made to work on Windows, but a
custom build of GCC was needed, with a couple of extra configure variables
set. Not sure if the proposed patch was incorporated into release versions
of GCC though.

See this messsage and its first response for a bit more info on it:

http://gcc.gnu.org/ml/gcc/2010-07/msg00069.html

Also, thought I'd give some instructions on how to invoke the plugin (not
that it works!), as a quick reference...

# checkout the repository
$ git clone git at github.com:alexleach/gccxml.git
$ cd gccxml

# switch to 'plugin' branch
$ git checkout plugin

# build debug version. This currently builds all of the patched GCC
backend too..
$ mkdir build && cd build
$ cmake config -DCMAKE_BUILD_TYPE=Debug ..
$ make -j6 gccxml_plugin

The built plugin will be in your build directory, at
./GCC/gcc/libgccxml_plugin.so

On Unix machines, add this folder to your LD_LIBRARY_PATH.
e.g.
$ export LD_LIBRARY_PATH="$PWD/GCC/gcc:$LD_LIBRARY_PATH"

On Windows, I guess change to the directory which contains the plugin.

# Load the gcccxml plugin, telling it to try and process one of the
provided GCC_XML test files.

g++ -fplugin=libgccxml_plugin.so
-fplugin-libgccxml_plugin-xml=TestUsualInclude.xml
../GCC_XML/Testing/TestUsualInclude.cxx
gccxml -fxml=TestUsualInclude.xml ../GCC_XML/Testing/TestUsualInclude.cxx

Note, both the above commands will need at least the following extra
include directories:
-IGCC_XML/GXFront -I../GCC_XML/GXFront -IGCC_XML


Cheers,
Alex

[Alex Leach]

On Sun, 10 Feb 2013 22:49:04 -0000, Andrej Mitrovic

Thanks for all this info, it will come in handy. It looks like the new
GCC has more dependencies than the old one (or maybe I just botched up
my dependencies since the last time I built gccxml). I'll need to get
my hands on MPFR, libcloog and others, but I'll try your plugin soon.
Thanks for the help.

I did add cmake checks for those libraries myself, so I've probably done
it a not particularly platform-agnostic manner... I found relevant
FindFoo.cmake scripts online, put them in GCC/config_cmake/ and called
them from GCC/config_cmake/CMakeLists.txt.

I already had those libraries installed (MPFR is part of the Gnu
Multi-Precision library, or GMP) , on a fairly fresh Arch Linux system
(built mid-December), so am not entirely sure if those packages are
strictly essential for GCC; there should be configure tests and override
switches for each of them.

To help and lead the upgrade process, I compared the files generated by
the native GCC-4.7.2 configure script to those generated by GCCXML's CMake
build system. The auto-host.h.in file needed a significant number of
changes; I put tests for its preprocessor macros all in that same
config_cmake/CMakeLists.txt file. No doubt there'll need to be some
further modifications to get that working on all platforms..

Cheers,
Alex

[Brad King]

Post by Alex Leach
Long story short, I thought I'd have a go at upgrading GCCXML's backend
compiler to GCC4.7.2. I made a fork on github a week ago and have been
working on it since.

Wonderful, I'm glad you've stepped forward to work on this!

I've finally had a chance to look through some of the work, though I'm
sorry I don't have time now to answer your questions.

I just made some changes to upstream gccxml to make this update easier.
First I re-tabified all the original GCC sources:

https://github.com/gccxml/gccxml/commit/2cbad55dd62f9c13cfab050ff53fc7cffc3515f1

Then I started an "upstream snapshot" branch that extracts the original
gcc 4.2.1 sources (like a vendor branch from CVS):

https://github.com/gccxml/gccxml/commit/de211e56086124ec676897fd2d779f9c85c773fe

and merged it in with "-s ours":

https://github.com/gccxml/gccxml/commit/b88a06a4e788723816d9e2db11cbdb74ac9e6e52

That will allow us to extend the upstream snapshot branch with a new
gcc version and merge it to bring in upstream changes.

I started a topic to do just such an upstream snapshot and merge,
published here:

https://github.com/bradking/gccxml/tree/update-gcc

I resolved the conflicts as described in the merge commit message
using your "re-apply" commit for guidance in a few cases (thanks!).
This approach preserves history cleanly.

Finally, I cherry-picked your build system updates onto the head
of the topic, but that is still a work in progress and I have not
tried to make the build work yet. Please fetch the topic and take
it from here if you have time. I don't know when I'll have time.

Post by Alex Leach
If you would be interested in merging any of it back into the main
repository, please let me know and I'll make a Pull Request. I think I
should get it working first, though...

If we can get it in good shape I'm interested, of course. This may
allow us to support newer compilers that make use of C++11 constructs
in their system headers.

Post by Alex Leach
One last thing worth mentioning. I had a look at the other fork, by Andrej
Mitrovic, and incorporated some of his feature additions, like putting C++
attributes into XML tags and detection of Pod, Anon and TypeScoped C++
Types.

Let's keep the gcc update topic separate from new features for now.

Thanks,
-Brad