URLEncode to UTF-8

🚧 Developers
1

I have some data I need to post via httpPost. Specifically, the data is x-www-form-urlencoded, and one of the values has special characters. As an example: a session key of "i50xj<502kdo20dkdow".

The < character needs to be URL encoded (utf-8). So < would be %3C. When I pass the data through URLEncode.encode(sessionKey, "utf-8"), the output is HTML encoded. So i50xj<502kdo20dkdow becomes i50xj%26lt%3B502kdo20dkdow instead of i50xj%3C502kdo20dkdow.

Any suggestions? Thanks!

2

Have you tried........

groovy.xml.XmlUtil.escapeXml(text)

3

It looks like I've traced this back a bit. The value I'm passing is specified as an application input, as such:

section {
    input "sessionKey", "password", title: "Private Session Key", required: true
}

When the value is stored, it's replacing the "<" with HTML encoded (<). How can I ensure app settings are stored as utf-8 strings?

4

You might not be able to prevent that unless you encode it first with & lt; or the other encoded character set representation.

You can tag a HE staff to see if the input controls have a preference to leave text alone but I haven't seen one.

Can you generate another key without invalid XML/HTTP characters?

5

Unfortunately, it's not a service I control, and I'm limited to the session keys they generate. For now, I've created an unHtmlValue routine within my app that will convert the < and > values back to their true values, but I'm surprised this is necessary.

6

Why? If you put characters in an HTML form that requires valid HTML characters it will encode the characters or fail to submit.

I'm not supposed by that at all. They can't cover every single use case on the first pass.

7

Because HTML is the interface, and its transport of data shouldn't be represented in storage. How are passwords handled then? Restricting characters because they're not "HTML Friendly" in a password field is very limited.

8

Illegal characters have to be encoded before they are sent over a web form. They may not even be doing it consciously. It could be some automatic part of some framework they're using. So, there's that. On the storage side if they just blindly un-escaped all text before saving it to the database then they would mess up my password which is:

i50xj&lt;502kdo20dkdow

Also, I get your dissatisfaction but removing three characters from the character set of hundreds or thousands (depending on which character set) does not make passwords "very limited" either.

@chuck.schwer Is there a possibility to get an input preference control flag that is aware of XML characters and correctly saves the value that is entered? Or is this technically a bug? How do the other guys do it? In an app if you save:

i50xj<502kdo20dkdow

The value is saved as the String above.

9

Its not a bug, it was due to a security audit.

https://community.hubitat.com/t/new-beta-release-2-1-0-available/15684

2 Likes
10

So, I was never implying this was a bug; At least once I realized characters were being replaced with HTML encoding. There are plenty of fields where this makes sense. But since I'm specifying an input type of "password", I would have assumed the data would not be subject to sanitization.

A mechanism for a developer to override this functionality would be nice. If that's not possible, can the value be encoded prior to being sanitized? (e.g. base64 encode before storage, and I would need to base64 decode before use)

For now, I will simply have to .replace(/</, "<") when the value is being used, which is easy enough. I appreciate the responses.