Eclipse Community Forums: Target Communication Framework » Debugger hang on _dl_debug

Home » Eclipse Projects » Target Communication Framework » Debugger hang on _dl_debug_state on linux/i.MX7(Issue with tcf-agent and Eclipse)

Show: Today's Messages :: Show Polls :: Message Navigator

Debugger hang on _dl_debug_state on linux/i.MX7 [message #1801041]

Fri, 11 January 2019 15:36

Brett Bergquist

Messages: 8
Registered: January 2019

Junior Member

I am having a tcf-agent/eclipse Target Communication Framework debug session hang when loading shared library. The target is embedded linux on an i.MX7 processor.

Many of the other shared libraries are loaded correctly but this one hangs. Suspending the thread that appears to be hanging I see within:

/* This function exists solely to have a breakpoint set on it by the
debugger. The debugger is supposed to find this function's address by
examining the r_brk member of struct r_debug, but GDB 4.15 in fact looks
for this particular symbol name in the PT_INTERP file. */
void
_dl_debug_state (void)
{
}

Here is what the stack trace looks like for the thread:

index.php/fa/34654/0/

If I detach the debugger, the process continues on normally and I can then re-attach and see that the application started up correctly.

Also if in Eclipse, I select a TCF agent and then use the Run Tests, the tests do not run successfully to this TCF agent. Here is a screen shot of what Eclipse reports:

index.php/fa/34655/0/

Any help on this will be greatly appreciated.

Attachment: Screen Shot 2019-01-11 at 10.06.35 AM.png
(Size: 69.42KB, Downloaded 325 times)
Attachment: Screen Shot 2019-01-11 at 10.34.22 AM.png
(Size: 156.08KB, Downloaded 310 times)

Report message to a moderator

Re: Debugger hang on _dl_debug_state on linux/i.MX7 [message #1801240 is a reply to message #1801041]

Tue, 15 January 2019 17:15

Eugene Tarassov

Messages: 17
Registered: July 2009

Junior Member

Hi Brett,

The stack trace indicates you are using Thumb instruction set for the libraries.
I'm afraid Thumb support is broken at the moment.
I'll work on a fix.

Regards,
Eugene

Report message to a moderator

Re: Debugger hang on _dl_debug_state on linux/i.MX7 [message #1801371 is a reply to message #1801041]

Thu, 17 January 2019 17:58

Eugene Tarassov

Messages: 17
Registered: July 2009

Junior Member

I have committed a fix for Thumb mode on ARM Linux.
It should work now.

Report message to a moderator

Re: Debugger hang on _dl_debug_state on linux/i.MX7 [message #1802776 is a reply to message #1801371]

Fri, 15 February 2019 14:09

Brett Bergquist

Messages: 8
Registered: January 2019

Junior Member

Wow I finally figured out that the issue was with thumb just today by rebuilding the cross compiler toolchain and playing with the switches until I found that when I had "-thumb" then the offsets of the breakpoint were wrong and then I did not they were correct and did research on thumb/arm and more searching and I found my own post here :). I usually hate when I find my own questions when searching but this time I am very happy :).

Thank you for taking the time to look at this and provide a fix. I spent there better part of a week on trying to determine why debugging worked on my Raspberry Pi and not on my i.MX7 projects and it turned out to be thumb/arm issue. In any case, I learned quite a bit on how the TCF agent works ;)

Report message to a moderator

Re: Debugger hang on _dl_debug_state on linux/i.MX7 [message #1802785 is a reply to message #1802776]

Fri, 15 February 2019 17:42

Brett Bergquist

Messages: 8
Registered: January 2019

Junior Member

I think there is still an issue with ARM/thumb and Linux and dynamically loading a library. When $loader_brk is evaluated after the application that will has the dynamic library load code is loaded, the Linux loader loads the address of the "_dl_debug_state" into there "_r_debug" structure's "rbrk" member. In this example, the information looks like:

0x76FFF8F0 00000001 76FFF908 76FE18D5 00000000 76FD6000 00000000 00000000 76FFFBF0 000F4F00 .....ùÿvÕ.þv.....`ýv........ðûÿv.O..

So the third word is supposed to be the address of "_dl_debug_state". Looking at the "_dl_debug_state" address, this is 0x76FE184D.

Looking at the runtime linker via "objdump -d ld-2.26.so" and looking for the "_dl_debug_state" we see:

0000b8d4 <_dl_debug_state@@GLIBC_PRIVATE>:

and if we look with "readelf -s ld-2.26.so" and looking for "_dl_debug_state" we see:

20: 0000b8d5 2 FUNC GLOBAL DEFAULT 10 _dl_debug_state@@GLIBC_PRIVATE

The TCF agent sets the breakpoint at address 76FE18D5 but "eventpoint_at_loader" is never hit and then the application being debugged is hung on the "dlopen". If I suspend the application, it is at the "_dl_debug_state" and this never continues. Disconnecting the debugger allows the application to continue and the "dlopen" succeeds.

Here is the simple stripped down application being debugged:

//#include <stdlib.h>
#include <stdio.h>
#include <dlfcn.h>

int main(int argc, char **argv) {
	printf("DynLoadLib started\n");
	void* handle = dlopen("libncurses.so.5", RTLD_LAZY);
	if (! handle) {
		printf("failed to load the library\n");
		return 1;
	}
	printf("library loaded at %x\n", handle);

	return 0;
}

I put in a hack in "cpu_bp_plant" where I look for the address 76FE18D5 and if found reset the address to 76FE18D4 (basically what is reported by "objdump" but with the virtual address) then the TCF correctly calls into "eventpoint_at_loader" and when continued the application being debugged correctly completes the "dlopen" and continues.

So it looks like something to do with ARM/THUMB here as well when determining the address to put the breakpoint on when evaluating "$loader_brk".

Any ideas on this?

Report message to a moderator

Re: Debugger hang on _dl_debug_state on linux/i.MX7 [message #1802788 is a reply to message #1802785]

Fri, 15 February 2019 21:26

Brett Bergquist

Messages: 8
Registered: January 2019

Junior Member

So I believe I know what is happening. When the hardware breakpoint is being set the address is being adjusted. In this case, the breakpoint address is 76FE18D5 but there hardware breakpoint is being set at 76FE18D4. The breakpoint is actually being hit but then in "cpu_bp_on_suspend", the breakpoint is not found since the address of the breakpoint does not match the actual address of the hardware breakpoint that was set.

So if the breakpoint address is adjusted, then either the original breakpoint address is going to need to be adjusted to match or whenever the comparison is done, then a check against the same adjusted address will need to be done.

Report message to a moderator

Re: Debugger hang on _dl_debug_state on linux/i.MX7 [message #1802826 is a reply to message #1802788]

Sun, 17 February 2019 15:57

Brett Bergquist

Messages: 8
Registered: January 2019

Junior Member

I changed the code with the attached patch and it fixes this specific issue but I am unsure if more is needed.

Basically if the hardware breakpoint address is being changed because it was an odd address and is altered to the even address, I update the original breakpoint address to be the same. When the breakpoint is hit and the callback is done all of the subsequent tests for a matching breakpoint address, etc. then work correctly.

Attachment: thumb-patch.diff
(Size: 1.56KB, Downloaded 61 times)

Report message to a moderator

Re: Debugger hang on _dl_debug_state on linux/i.MX7 [message #1802828 is a reply to message #1802826]

Sun, 17 February 2019 17:25

Eugene Tarassov

Messages: 17
Registered: July 2009

Junior Member

Good catch!
On ARM, function pointer can have bit 0 set to 1 to indicate Thumb code.
The bit needs to be cleared to turn function pointer to a valid breakpoint address.
Actually, I already have submitted a patch for this, but it takes ~2 days for a patch to complete all regression tests.
It is available now.
Thanks!

Report message to a moderator

Re: Debugger hang on _dl_debug_state on linux/i.MX7 [message #1802896 is a reply to message #1802828]

Mon, 18 February 2019 16:58

Brett Bergquist

Messages: 8
Registered: January 2019

Junior Member

I built with your change and it is working. I was wondering about other breakpoint targets that might be in Thumb mode, for example if I have a function that is has been compiled to a Thumb address (ie 1 in the least significant bit) and I set a function breakpoint, I was concerned that the address would be adjusted and checked in "cpudefs-mdep.c" but was wondering if anything special needed to be done like you did in "context-linux.c" for the "r_debug.r_brk". I tried a test case and it did seem to work okay but was wondering why the special case is needed for "r_debug.r_brk" and not other breakpoint address type lookups?

Just for my own curiosity.

I appreciate your time on this. BTW, I work with Chris Wyse who remembers you from his time a Wind River. When I mentioned I had posted to the forum he said, "Who responded to this? Eugene Tarassov? :)

Report message to a moderator

Previous Topic:	TCF / Linux error signal 11 ?
Next Topic:	no TCF target in Connection list

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Fri Dec 04 15:36:54 GMT 2020

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter