• Re: Ref-strings in logging messages (was: Performance issue with CPytho

    From Weatherby,Gerard@21:1/5 to All on Sat Oct 8 10:50:51 2022
    TG9nZ2luZyBkb2VzIHN1cHBvcnQgcGFzc2luZyBhIGNhbGxhYmxlLCBpZiBpbmRpcmVjdGx5LiBJ dCBvbmx5IGNhbGxzIF9fc3RyX18gb24gdGhlIG9iamVjdCBwYXNzZWQgaWYgZGVidWdnaW5nIGlz IGVuYWJsZWQuDQoNCmNsYXNzIERlZmVyOg0KDQogICAgZGVmIF9faW5pdF9fKHNlbGYsZm4pOg0K ICAgICAgICBzZWxmLmZuID0gZm4NCg0KICAgIGRlZiBfX3N0cl9fKHNlbGYpOg0KICAgICAgICBy ZXR1cm4gc2VsZi5mbigpDQoNCmRlZiBzb21lX2V4cGVuc2l2ZV9mdW5jdGlvbigpOg0KICAgIHJl dHVybiAiaGVsbG8iDQoNCmxvZ2dpbmcuYmFzaWNDb25maWcoKQ0KbG9nZ2luZy5kZWJ1ZyhEZWZl cihzb21lX2V4cGVuc2l2ZV9mdW5jdGlvbikpDQoNCg0KRnJvbTogUHl0aG9uLWxpc3QgPHB5dGhv bi1saXN0LWJvdW5jZXMrZ3dlYXRoZXJieT11Y2hjLmVkdUBweXRob24ub3JnPiBvbiBiZWhhbGYg b2YgQmFycnkgPGJhcnJ5QGJhcnJ5cy1lbWFjcy5vcmc+DQpEYXRlOiBGcmlkYXksIE9jdG9iZXIg NywgMjAyMiBhdCAxOjMwIFBNDQpUbzogTVJBQiA8cHl0aG9uQG1yYWJhcm5ldHQucGx1cy5jb20+ DQpDYzogcHl0aG9uLWxpc3RAcHl0aG9uLm9yZyA8cHl0aG9uLWxpc3RAcHl0aG9uLm9yZz4NClN1 YmplY3Q6IFJlOiBSZWYtc3RyaW5ncyBpbiBsb2dnaW5nIG1lc3NhZ2VzICh3YXM6IFBlcmZvcm1h bmNlIGlzc3VlIHdpdGggQ1B5dGhvbiAzLjEwICsgQ3l0aG9uKQ0KKioqIEF0dGVudGlvbjogVGhp cyBpcyBhbiBleHRlcm5hbCBlbWFpbC4gVXNlIGNhdXRpb24gcmVzcG9uZGluZywgb3BlbmluZyBh dHRhY2htZW50cyBvciBjbGlja2luZyBvbiBsaW5rcy4gKioqDQoNCj4gT24gNyBPY3QgMjAyMiwg YXQgMTg6MTYsIE1SQUIgPHB5dGhvbkBtcmFiYXJuZXR0LnBsdXMuY29tPiB3cm90ZToNCj4NCj4g 77u/T24gMjAyMi0xMC0wNyAxNjo0NSwgU2tpcCBNb250YW5hcm8gd3JvdGU6DQo+Pj4gT24gRnJp LCBPY3QgNywgMjAyMiBhdCA5OjQyIEFNIEFuZHJlYXMgQW1lcyA8YW5kcmVhcy4wODE1LnF3ZXJ0 ekBnbWFpbC5jb20+DQo+Pj4gd3JvdGU6DQo+Pj4gMS4gVGhlIGN1bHByaXQgd2FzIG1lLiBBcyBs YXp5IGFzIEkgYW0sIEkgaGF2ZSB1c2VkIGYtc3RyaW5ncyBhbGwgb3ZlciB0aGUNCj4+PiBwbGFj ZSBpbiBjYWxscyB0byBgbG9nZ2luZy5sb2dnZXIuZGVidWcoKWAgYW5kIGZyaWVuZHMsIGV2YWx1 YXRpbmcgYWxsDQo+Pj4gYXJndW1lbnRzIHJlZ2FyZGxlc3Mgb2Ygd2hldGhlciB0aGUgbG9nZ2Vy IHdhcyBlbmFibGVkIG9yIG5vdC4NCj4+Pg0KPj4gSSB0aG91Z2h0IHRoZXJlIHdhcyBzb21lIGRp c2N1c3Npb24gYWJvdXQgd2hldGhlciBhbmQgaG93IHRvIGVmZmljaWVudGx5DQo+PiBhZG1pdCBm LXN0cmluZ3MgdG8gdGhlIGxvZ2dpbmcgcGFja2FnZS4gSSdtIGd1ZXNzaW5nIHRoYXQncyBub3Qg Z29uZQ0KPj4gYW55d2hlcmUgKHlldCkuDQo+IExldHRpbmcgeW91IHBhc3MgaW4gYSBjYWxsYWJs ZSB0byBjYWxsIG1pZ2h0IGhlbHAgYmVjYXVzZSB0aGF0IHlvdSBjb3VsZCB1c2UgbGFtYmRhLg0K DQpZZXAsIHRoYXTigJlzIHRoZSBvYnZpb3VzIHdheSB0byBhdm9pZCBleHBlbnNpdmUgbG9nIGRh dGEgZ2VuZXJhdGlvbi4NCldvdWxkIG5lZWQgbG9nZ2luZyBtb2R1bGUgdG8gc3VwcG9ydCB0aGF0 IHVzZSBjYXNlLg0KDQpCYXJyeQ0KDQo+IC0tDQo+IGh0dHBzOi8vdXJsZGVmZW5zZS5jb20vdjMv X19odHRwczovL21haWwucHl0aG9uLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3B5dGhvbi1saXN0X187 ISFDbl9VWF9wMyFtckVTeEFqOVlDSHNkdE5BZmtOaVktWmY2VTNXVElxYU5yZ0JtYncxRUxsUXk1 MWlsb2I0M2REME9Oc3F2ZzRhOTRNRWRPZHdvbWd5cWZ5QUJidlJuQSQ8aHR0cHM6Ly91cmxkZWZl bnNlLmNvbS92My9fX2h0dHBzOi9tYWlsLnB5dGhvbi5vcmcvbWFpbG1hbi9saXN0aW5mby9weXRo b24tbGlzdF9fOyEhQ25fVVhfcDMhbXJFU3hBajlZQ0hzZHROQWZrTmlZLVpmNlUzV1RJcWFOcmdC bWJ3MUVMbFF5NTFpbG9iNDNkRDBPTnNxdmc0YTk0TUVkT2R3b21neXFmeUFCYnZSbkEkPg0KPg0K DQotLQ0KaHR0cHM6Ly91cmxkZWZlbnNlLmNvbS92My9fX2h0dHBzOi8vbWFpbC5weXRob24ub3Jn L21haWxtYW4vbGlzdGluZm8vcHl0aG9uLWxpc3RfXzshIUNuX1VYX3AzIW1yRVN4QWo5WUNIc2R0 TkFma05pWS1aZjZVM1dUSXFhTnJnQm1idzFFTGxReTUxaWxvYjQzZEQwT05zcXZnNGE5NE1FZE9k d29tZ3lxZnlBQmJ2Um5BJDxodHRwczovL3VybGRlZmVuc2UuY29tL3YzL19faHR0cHM6L21haWwu cHl0aG9uLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3B5dGhvbi1saXN0X187ISFDbl9VWF9wMyFtckVT eEFqOVlDSHNkdE5BZmtOaVktWmY2VTNXVElxYU5yZ0JtYncxRUxsUXk1MWlsb2I0M2REME9Oc3F2 ZzRhOTRNRWRPZHdvbWd5cWZ5QUJidlJuQSQ+DQo=

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Barry Scott@21:1/5 to All on Mon Oct 24 21:52:12 2022
    On 8 Oct 2022, at 11:50, Weatherby,Gerard <[email protected]> wrote:

    Logging does support passing a callable, if indirectly. It only calls __str__ on the object passed if debugging is enabled.

    class Defer:

    def __init__(self,fn):
    self.fn = fn

    def __str__(self):
    return self.fn()

    def some_expensive_function():
    return "hello"

    logging.basicConfig()
    logging.debug(Defer(some_expensive_function))

    Oh what a clever hack. Took a few minutes of code reading to see why this works.
    You are exploiting the str(msg) that is in class LogRecords getMessage().

    ```
    def getMessage(self):
    """
    Return the message for this LogRecord.

    Return the message for this LogRecord after merging any user-supplied
    arguments with the message.
    """
    msg = str(self.msg)
    if self.args:
    msg = msg % self.args
    return msg
    ```

    Barry




    From: Python-list <python-list-bounces+gweatherby=[email protected] <mailto:python-list-bounces+gweatherby=[email protected]>> on behalf of Barry <[email protected] <mailto:[email protected]>>
    Date: Friday, October 7, 2022 at 1:30 PM
    To: MRAB <[email protected] <mailto:[email protected]>>
    Cc: [email protected] <mailto:[email protected]> <[email protected] <mailto:[email protected]>>
    Subject: Re: Ref-strings in logging messages (was: Performance issue with CPython 3.10 + Cython)

    *** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***

    On 7 Oct 2022, at 18:16, MRAB <[email protected]> wrote:

    On 2022-10-07 16:45, Skip Montanaro wrote:
    On Fri, Oct 7, 2022 at 9:42 AM Andreas Ames <[email protected]>
    wrote:
    1. The culprit was me. As lazy as I am, I have used f-strings all over the
    place in calls to `logging.logger.debug()` and friends, evaluating all >>> arguments regardless of whether the logger was enabled or not.

    I thought there was some discussion about whether and how to efficiently >> admit f-strings to the logging package. I'm guessing that's not gone
    anywhere (yet).
    Letting you pass in a callable to call might help because that you could use lambda.

    Yep, that’s the obvious way to avoid expensive log data generation.
    Would need logging module to support that use case.

    Barry

    -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!mrESxAj9YCHsdtNAfkNiY-Zf6U3WTIqaNrgBmbw1ELlQy51ilob43dD0ONsqvg4a94MEdOdwomgyqfyABbvRnA$ <https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/
    python-list__;!!Cn_UX_p3!mrESxAj9YCHsdtNAfkNiY-Zf6U3WTIqaNrgBmbw1ELlQy51ilob43dD0ONsqvg4a94MEdOdwomgyqfyABbvRnA$>


    -- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!mrESxAj9YCHsdtNAfkNiY-Zf6U3WTIqaNrgBmbw1ELlQy51ilob43dD0ONsqvg4a94MEdOdwomgyqfyABbvRnA$ <https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/
    python-list__;!!Cn_UX_p3!mrESxAj9YCHsdtNAfkNiY-Zf6U3WTIqaNrgBmbw1ELlQy51ilob43dD0ONsqvg4a94MEdOdwomgyqfyABbvRnA$>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)